2024-04-27T00:55:27.325 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-27T00:55:27.326 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} --archive /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458 --owner scheduled_teuthology@teuthology --name teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458/orig.config.yaml 2024-04-27T00:55:27.359 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T00:55:27.433 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458 branch: squid description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7675458' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug ms: 1 global: mon election default strategy: 3 ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) log-only-match: - CEPHADM_ mon_bind_msgr2: false sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a extra_packages: - cephadm selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 4854 sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a sleep_before_teardown: 0 subset: 40/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a targets: smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYtc3gmzsSXF1EWR8wMl0Oy/Tt/H+LWCodmdZt8vaT64m7TNLtkO530iQU1yb2c4PuuruJ2FpuCARswI35OQ7E= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAkXHwIsYg0oR0B5M0CtwPH/fwsp++rfgLJ+MKs+6qqoIRMW1omHKWzLudLbLstUmrGA7YsqQ7U+6LO0ROYS7Cw= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_21:08:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-27T00:55:27.433 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa; will attempt to use it 2024-04-27T00:55:27.434 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks 2024-04-27T00:55:27.434 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-27T00:55:27.435 INFO:teuthology.task.internal:Checking packages... 2024-04-27T00:55:27.461 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a' 2024-04-27T00:55:27.461 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T00:55:27.461 INFO:teuthology.packaging:ref: None 2024-04-27T00:55:27.461 INFO:teuthology.packaging:tag: None 2024-04-27T00:55:27.461 INFO:teuthology.packaging:branch: squid 2024-04-27T00:55:27.461 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T00:55:27.462 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-27T00:55:27.769 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2447.gb22e2ebd 2024-04-27T00:55:27.770 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-27T00:55:27.777 INFO:teuthology.task.internal:no buildpackages task found 2024-04-27T00:55:27.777 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-27T00:55:27.818 INFO:teuthology.task.internal:Saving configuration 2024-04-27T00:55:27.829 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-27T00:55:27.851 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T00:55:27.873 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 00:47:35.003827', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPYtc3gmzsSXF1EWR8wMl0Oy/Tt/H+LWCodmdZt8vaT64m7TNLtkO530iQU1yb2c4PuuruJ2FpuCARswI35OQ7E='} 2024-04-27T00:55:27.893 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 00:47:35.004998', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAkXHwIsYg0oR0B5M0CtwPH/fwsp++rfgLJ+MKs+6qqoIRMW1omHKWzLudLbLstUmrGA7YsqQ7U+6LO0ROYS7Cw='} 2024-04-27T00:55:27.894 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-27T00:55:27.900 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-04-27T00:55:27.900 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-04-27T00:55:27.900 INFO:teuthology.run_tasks:Running task console_log... 2024-04-27T00:55:27.989 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f66de791f70>, signals=[15]) 2024-04-27T00:55:27.990 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-27T00:55:27.998 INFO:teuthology.task.internal:Opening connections... 2024-04-27T00:55:27.998 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-04-27T00:55:28.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T00:55:28.081 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-04-27T00:55:28.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T00:55:28.156 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-27T00:55:28.163 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-04-27T00:55:28.181 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-04-27T00:55:28.182 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-04-27T00:55:28.238 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-04-27T00:55:28.238 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-04-27T00:55:28.238 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-04-27T00:55:28.238 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-04-27T00:55:28.238 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-04-27T00:55:28.238 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-04-27T00:55:28.239 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-27T00:55:28.239 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-04-27T00:55:28.239 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-04-27T00:55:28.239 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-27T00:55:28.239 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-04-27T00:55:28.239 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T00:55:28.239 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-27T00:55:28.239 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-27T00:55:28.240 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-04-27T00:55:28.261 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-04-27T00:55:28.280 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-04-27T00:55:28.280 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-04-27T00:55:28.337 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-04-27T00:55:28.337 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-04-27T00:55:28.337 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-04-27T00:55:28.337 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-04-27T00:55:28.337 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-04-27T00:55:28.337 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-04-27T00:55:28.337 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-27T00:55:28.337 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-04-27T00:55:28.337 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-04-27T00:55:28.338 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-27T00:55:28.338 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-04-27T00:55:28.338 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-27T00:55:28.338 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-27T00:55:28.338 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-27T00:55:28.338 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-04-27T00:55:28.360 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-27T00:55:28.370 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-27T00:55:28.392 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-27T00:55:28.392 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-04-27T00:55:28.395 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-04-27T00:55:28.411 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-27T00:55:28.417 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-27T00:55:28.417 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-04-27T00:55:28.452 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-04-27T00:55:28.553 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-27T00:55:28.608 INFO:teuthology.run_tasks:Running task kernel... 2024-04-27T00:55:28.635 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-27T00:55:28.635 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-27T00:55:28.636 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T00:55:28.636 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T00:55:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:55:28.653 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-04-27T00:55:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:55:28.654 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-04-27T00:55:28.710 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-437.el9.x86_64 2024-04-27T00:55:28.710 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-437.el9.x86_64 2024-04-27T00:55:28.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-04-27T00:55:28.711 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-437.el9.x86_64 2024-04-27T00:55:28.711 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-437.el9.x86_64 2024-04-27T00:55:28.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-04-27T00:55:31.145 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-27T00:55:32.609 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-04-27T00:55:34.870 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-27T00:55:35.270 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 19 MB 00:03 2024-04-27T00:55:46.741 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 765 kB/s | 6.2 MB 00:08 2024-04-27T00:55:47.094 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 779 kB/s | 6.2 MB 00:08 2024-04-27T00:55:48.359 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-04-27T00:55:48.945 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-27T00:55:50.037 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-27T00:55:50.046 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-27T00:55:55.118 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-27T00:55:55.162 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-04-27T00:55:56.966 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-27T00:55:56.966 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-27T00:55:56.993 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-27T00:55:56.993 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-27T00:55:57.051 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T00:55:57.052 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-27T00:55:57.052 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T00:55:57.088 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T00:55:57.089 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-04-27T00:55:57.089 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T00:55:57.151 DEBUG:teuthology.orchestra.run.smithi181:> echo no | sudo yum reinstall kernel || true 2024-04-27T00:55:57.216 DEBUG:teuthology.orchestra.run.smithi162:> echo no | sudo yum reinstall kernel || true 2024-04-27T00:55:57.647 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:02 ago on Sat 27 Apr 2024 12:55:55 AM UTC. 2024-04-27T00:55:57.720 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:02 ago on Sat 27 Apr 2024 12:55:55 AM UTC. 2024-04-27T00:55:57.773 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T00:55:57.785 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T00:55:57.785 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-04-27T00:55:57.785 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T00:55:57.785 INFO:teuthology.orchestra.run.smithi181.stdout:Reinstalling: 2024-04-27T00:55:57.785 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T00:55:57.786 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T00:55:57.786 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T00:55:57.786 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T00:55:57.786 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T00:55:57.786 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 255 k 2024-04-27T00:55:57.786 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 0 2024-04-27T00:55:57.786 INFO:teuthology.orchestra.run.smithi181.stderr:Operation aborted. 2024-04-27T00:55:57.838 INFO:teuthology.orchestra.run.smithi181.stdout:Is this ok [y/N]: 2024-04-27T00:55:57.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum reinstall -y kernel || true 2024-04-27T00:55:57.849 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T00:55:57.855 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T00:55:57.855 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-04-27T00:55:57.855 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T00:55:57.855 INFO:teuthology.orchestra.run.smithi162.stdout:Reinstalling: 2024-04-27T00:55:57.855 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T00:55:57.855 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T00:55:57.856 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T00:55:57.856 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T00:55:57.856 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T00:55:57.856 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 255 k 2024-04-27T00:55:57.856 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 0 2024-04-27T00:55:57.856 INFO:teuthology.orchestra.run.smithi162.stderr:Operation aborted. 2024-04-27T00:55:57.922 INFO:teuthology.orchestra.run.smithi162.stdout:Is this ok [y/N]: 2024-04-27T00:55:57.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum reinstall -y kernel || true 2024-04-27T00:55:58.332 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 12:55:55 AM UTC. 2024-04-27T00:55:58.418 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:03 ago on Sat 27 Apr 2024 12:55:55 AM UTC. 2024-04-27T00:55:58.447 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T00:55:58.448 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T00:55:58.448 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-04-27T00:55:58.448 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T00:55:58.448 INFO:teuthology.orchestra.run.smithi181.stdout:Reinstalling: 2024-04-27T00:55:58.448 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T00:55:58.449 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T00:55:58.449 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T00:55:58.449 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T00:55:58.449 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T00:55:58.449 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 255 k 2024-04-27T00:55:58.449 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 0 2024-04-27T00:55:58.449 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T00:55:58.535 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout:Reinstalling: 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 255 k 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 0 2024-04-27T00:55:58.536 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T00:55:58.670 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.6 MB/s | 255 kB 00:00 2024-04-27T00:55:58.670 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T00:55:58.671 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.1 MB/s | 255 kB 00:00 2024-04-27T00:55:58.690 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T00:55:58.874 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T00:55:58.874 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T00:55:58.879 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T00:55:58.880 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T00:55:59.027 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T00:55:59.105 INFO:teuthology.orchestra.run.smithi181.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T00:55:59.105 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:55:59.555 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-437.el9.x86_64.rpm 269 kB/s | 255 kB 00:00 2024-04-27T00:55:59.556 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T00:55:59.556 INFO:teuthology.orchestra.run.smithi162.stdout:Total 250 kB/s | 255 kB 00:01 2024-04-27T00:55:59.576 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T00:55:59.744 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:55:59.744 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T00:55:59.750 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T00:55:59.750 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T00:55:59.755 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T00:55:59.756 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T00:55:59.900 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T00:55:59.965 INFO:teuthology.orchestra.run.smithi162.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T00:55:59.965 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:56:00.432 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:56:00.432 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T00:56:00.432 INFO:teuthology.orchestra.run.smithi181.stdout:Reinstalled: 2024-04-27T00:56:00.432 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-27T00:56:00.433 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T00:56:00.433 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T00:56:00.602 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:56:00.602 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-27T00:56:00.619 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T00:56:00.665 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-27T00:56:00.667 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-27T00:56:00.667 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T00:56:00.667 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-27T00:56:00.667 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T00:56:00.700 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-04-27T00:56:00.729 DEBUG:teuthology.parallel:result is None 2024-04-27T00:56:01.306 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-27T00:56:01.306 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T00:56:01.306 INFO:teuthology.orchestra.run.smithi162.stdout:Reinstalled: 2024-04-27T00:56:01.306 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-27T00:56:01.306 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T00:56:01.306 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T00:56:01.483 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-04-27T00:56:01.530 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-27T00:56:01.530 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-27T00:56:01.530 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-27T00:56:01.530 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-27T00:56:01.530 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T00:56:01.559 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-04-27T00:56:01.581 DEBUG:teuthology.parallel:result is None 2024-04-27T00:56:01.581 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-27T00:56:01.589 INFO:teuthology.task.internal:Creating test directory... 2024-04-27T00:56:01.590 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T00:56:01.623 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T00:56:01.640 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-27T00:56:01.646 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-27T00:56:01.672 INFO:teuthology.task.internal:Creating archive directory... 2024-04-27T00:56:01.672 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T00:56:01.678 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T00:56:01.708 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-27T00:56:01.715 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-27T00:56:01.715 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T00:56:01.744 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T00:56:01.775 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T00:56:01.783 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T00:56:01.784 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T00:56:01.792 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T00:56:01.794 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-27T00:56:01.801 INFO:teuthology.task.internal:Configuring sudo... 2024-04-27T00:56:01.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T00:56:01.828 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T00:56:01.858 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-27T00:56:01.868 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-27T00:56:01.868 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T00:56:01.893 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T00:56:01.915 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T00:56:01.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T00:56:02.034 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T00:56:02.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T00:56:02.154 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T00:56:02.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T00:56:02.218 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T00:56:02.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T00:56:02.311 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T00:56:02.394 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T00:56:02.431 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T00:56:02.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T00:56:02.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-04-27T00:56:02.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-04-27T00:56:02.532 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T00:56:02.575 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T00:56:03.061 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-27T00:56:03.069 INFO:teuthology.task.internal:Starting timer... 2024-04-27T00:56:03.070 INFO:teuthology.run_tasks:Running task pcp... 2024-04-27T00:56:03.125 INFO:teuthology.run_tasks:Running task selinux... 2024-04-27T00:56:03.175 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-27T00:56:03.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-04-27T00:56:03.256 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-04-27T00:56:03.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-04-27T00:56:03.311 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-04-27T00:56:03.314 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-27T00:56:03.314 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-04-27T00:56:03.335 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-04-27T00:56:03.336 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-04-27T00:56:03.353 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-04-27T00:56:03.354 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi162.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-04-27T00:56:03.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-27T00:56:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:56:03.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-27T00:56:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:56:03.424 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-27T00:56:03.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-04-27T00:56:03.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-04-27T00:56:03.522 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-27T00:56:03.532 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-27T00:56:03.553 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-27T00:56:03.555 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi162.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-27T00:59:39.912 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi162.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-04-27T00:59:39.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-04-27T00:59:39.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T00:59:39.989 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-04-27T00:59:40.065 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-04-27T00:59:40.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-27T00:59:40.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T00:59:40.138 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-04-27T00:59:40.217 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-27T00:59:40.217 INFO:teuthology.run_tasks:Running task clock... 2024-04-27T00:59:40.230 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-27T00:59:40.230 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T00:59:40.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T00:59:40.233 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T00:59:40.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T00:59:40.258 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T00:59:40.272 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T00:59:40.284 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-27T00:59:40.294 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-04-27T00:59:40.297 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-27T00:59:40.305 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-04-27T00:59:40.318 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T00:59:40.319 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-04-27T00:59:40.330 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-04-27T00:59:40.331 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T00:59:40.343 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-27T00:59:40.355 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-27T00:59:40.383 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-04-27T00:59:40.386 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T00:59:40.386 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-04-27T00:59:40.386 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:59:40.386 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:59:40.386 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:59:40.386 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:59:40.412 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-04-27T00:59:40.416 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T00:59:40.416 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-04-27T00:59:40.416 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:59:40.416 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:59:40.416 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:59:40.416 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:59:40.417 INFO:teuthology.run_tasks:Running task install... 2024-04-27T00:59:40.425 DEBUG:teuthology.task.install:project ceph 2024-04-27T00:59:40.425 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}, 'extra_packages': ['cephadm']} 2024-04-27T00:59:40.425 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'} 2024-04-27T00:59:40.426 INFO:teuthology.task.install:Using flavor: default 2024-04-27T00:59:40.432 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-27T00:59:40.432 INFO:teuthology.task.install:extra packages: [] 2024-04-27T00:59:40.432 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'tag': None, 'wait_for_package': False} 2024-04-27T00:59:40.433 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T00:59:40.435 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'tag': None, 'wait_for_package': False} 2024-04-27T00:59:40.435 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T00:59:40.577 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/centos/9/flavors/default/ 2024-04-27T00:59:40.577 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2447.gb22e2ebd 2024-04-27T00:59:40.652 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/centos/9/flavors/default/ 2024-04-27T00:59:40.653 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2447.gb22e2ebd 2024-04-27T00:59:40.799 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-27T00:59:40.799 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T00:59:40.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-27T00:59:40.823 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-27T00:59:40.824 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-27T00:59:40.888 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-27T00:59:40.961 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-27T00:59:40.961 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T00:59:40.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-27T00:59:40.962 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-27T00:59:40.990 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-27T00:59:40.991 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-27T00:59:41.025 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-04-27T00:59:41.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-04-27T00:59:41.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-27T00:59:41.131 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-27T00:59:41.195 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-04-27T00:59:41.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-04-27T00:59:41.248 INFO:teuthology.orchestra.run.smithi181.stdout:48 files removed 2024-04-27T00:59:41.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-04-27T00:59:41.420 INFO:teuthology.orchestra.run.smithi162.stdout:48 files removed 2024-04-27T00:59:41.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw 2024-04-27T00:59:42.159 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 132 kB/s | 83 kB 00:00 2024-04-27T00:59:42.195 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 170 kB/s | 83 kB 00:00 2024-04-27T00:59:42.570 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-04-27T00:59:42.585 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-04-27T00:59:42.979 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-27T00:59:42.985 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-04-27T00:59:44.688 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-04-27T00:59:46.774 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-04-27T00:59:50.964 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.2 MB 00:00 2024-04-27T00:59:51.835 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 935 kB/s | 8.0 MB 00:08 2024-04-27T00:59:52.731 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-04-27T00:59:54.066 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-27T00:59:59.157 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-27T01:00:01.331 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:00:01.333 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:01.333 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-04-27T01:00:01.333 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:01.333 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 10 M 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 3.4 M 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 3.1 M 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 5.4 M 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 21 M 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 25 k 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-27T01:00:01.334 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 745 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 501 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 5.3 M 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 45 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 133 k 2024-04-27T01:00:01.335 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 161 k 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 321 k 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 298 k 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 99 k 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:00:01.336 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:01.337 INFO:teuthology.orchestra.run.smithi162.stdout:Install 30 Packages 2024-04-27T01:00:01.337 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 2 Packages 2024-04-27T01:00:01.337 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:01.337 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 60 M 2024-04-27T01:00:01.337 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:00:02.841 INFO:teuthology.orchestra.run.smithi162.stdout:(1/32): ceph-base-19.0.0-2447.gb22e2ebd.el9.x86 4.9 MB/s | 5.4 MB 00:01 2024-04-27T01:00:02.899 INFO:teuthology.orchestra.run.smithi162.stdout:(2/32): ceph-selinux-19.0.0-2447.gb22e2ebd.el9. 433 kB/s | 25 kB 00:00 2024-04-27T01:00:03.008 INFO:teuthology.orchestra.run.smithi162.stdout:(3/32): libcephfs2-19.0.0-2447.gb22e2ebd.el9.x8 6.7 MB/s | 745 kB 00:00 2024-04-27T01:00:03.091 INFO:teuthology.orchestra.run.smithi162.stdout:(4/32): libradosstriper1-19.0.0-2447.gb22e2ebd. 5.9 MB/s | 501 kB 00:00 2024-04-27T01:00:03.342 INFO:teuthology.orchestra.run.smithi162.stdout:(5/32): ceph-radosgw-19.0.0-2447.gb22e2ebd.el9. 6.4 MB/s | 10 MB 00:01 2024-04-27T01:00:03.383 INFO:teuthology.orchestra.run.smithi162.stdout:(6/32): python3-ceph-argparse-19.0.0-2447.gb22e 1.1 MB/s | 45 kB 00:00 2024-04-27T01:00:03.442 INFO:teuthology.orchestra.run.smithi162.stdout:(7/32): python3-ceph-common-19.0.0-2447.gb22e2e 2.2 MB/s | 133 kB 00:00 2024-04-27T01:00:03.500 INFO:teuthology.orchestra.run.smithi162.stdout:(8/32): python3-cephfs-19.0.0-2447.gb22e2ebd.el 2.7 MB/s | 161 kB 00:00 2024-04-27T01:00:03.568 INFO:teuthology.orchestra.run.smithi162.stdout:(9/32): python3-rados-19.0.0-2447.gb22e2ebd.el9 4.7 MB/s | 321 kB 00:00 2024-04-27T01:00:03.693 INFO:teuthology.orchestra.run.smithi162.stdout:(10/32): librgw2-19.0.0-2447.gb22e2ebd.el9.x86_ 8.8 MB/s | 5.3 MB 00:00 2024-04-27T01:00:03.718 INFO:teuthology.orchestra.run.smithi162.stdout:(11/32): python3-rbd-19.0.0-2447.gb22e2ebd.el9. 1.9 MB/s | 298 kB 00:00 2024-04-27T01:00:03.743 INFO:teuthology.orchestra.run.smithi162.stdout:(12/32): python3-rgw-19.0.0-2447.gb22e2ebd.el9. 1.9 MB/s | 99 kB 00:00 2024-04-27T01:00:04.110 INFO:teuthology.orchestra.run.smithi162.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 273 kB/s | 107 kB 00:00 2024-04-27T01:00:04.176 INFO:teuthology.orchestra.run.smithi162.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 370 kB/s | 160 kB 00:00 2024-04-27T01:00:04.219 INFO:teuthology.orchestra.run.smithi162.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-04-27T01:00:04.278 INFO:teuthology.orchestra.run.smithi162.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-04-27T01:00:04.336 INFO:teuthology.orchestra.run.smithi162.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-27T01:00:04.362 INFO:teuthology.orchestra.run.smithi162.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 504 kB/s | 42 kB 00:00 2024-04-27T01:00:04.420 INFO:teuthology.orchestra.run.smithi162.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 619 kB/s | 36 kB 00:00 2024-04-27T01:00:04.445 INFO:teuthology.orchestra.run.smithi162.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-27T01:00:04.496 INFO:teuthology.orchestra.run.smithi162.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 624 kB/s | 47 kB 00:00 2024-04-27T01:00:04.705 INFO:teuthology.orchestra.run.smithi162.stdout:(22/32): ceph-common-19.0.0-2447.gb22e2ebd.el9. 7.1 MB/s | 21 MB 00:02 2024-04-27T01:00:04.955 INFO:teuthology.orchestra.run.smithi162.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 104 kB/s | 26 kB 00:00 2024-04-27T01:00:05.139 INFO:teuthology.orchestra.run.smithi162.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 265 kB/s | 49 kB 00:00 2024-04-27T01:00:05.172 INFO:teuthology.orchestra.run.smithi162.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 425 kB/s | 309 kB 00:00 2024-04-27T01:00:05.339 INFO:teuthology.orchestra.run.smithi162.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 337 kB/s | 67 kB 00:00 2024-04-27T01:00:05.681 INFO:teuthology.orchestra.run.smithi162.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.6 MB/s | 839 kB 00:00 2024-04-27T01:00:05.731 INFO:teuthology.orchestra.run.smithi162.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 489 kB/s | 192 kB 00:00 2024-04-27T01:00:05.800 INFO:teuthology.orchestra.run.smithi162.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-04-27T01:00:06.107 INFO:teuthology.orchestra.run.smithi162.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 3.7 MB/s | 1.6 MB 00:00 2024-04-27T01:00:06.357 INFO:teuthology.orchestra.run.smithi162.stdout:(31/32): librados2-19.0.0-2447.gb22e2ebd.el9.x8 5.4 MB/s | 3.4 MB 00:00 2024-04-27T01:00:06.457 INFO:teuthology.orchestra.run.smithi162.stdout:(32/32): librbd1-19.0.0-2447.gb22e2ebd.el9.x86_ 4.8 MB/s | 3.1 MB 00:00 2024-04-27T01:00:06.461 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:00:06.461 INFO:teuthology.orchestra.run.smithi162.stdout:Total 12 MB/s | 60 MB 00:05 2024-04-27T01:00:06.562 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:00:06.599 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:00:06.599 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:00:07.062 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:00:07.063 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:00:08.573 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:00:08.665 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-27T01:00:08.694 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-27T01:00:08.720 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-27T01:00:08.743 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-27T01:00:08.779 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-27T01:00:08.935 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-27T01:00:08.989 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-27T01:00:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-27T01:00:09.439 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 7/34 2024-04-27T01:00:09.515 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 7/34 2024-04-27T01:00:09.612 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.0.0-2447.gb22e2ebd.el9.x86_64 8/34 2024-04-27T01:00:09.649 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 9/34 2024-04-27T01:00:09.711 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 9/34 2024-04-27T01:00:09.748 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-27T01:00:09.792 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.0.0-2447.gb22e2ebd.el9. 11/34 2024-04-27T01:00:09.861 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.0.0-2447.gb22e2ebd.el9.x86_64 12/34 2024-04-27T01:00:09.884 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.0.0-2447.gb22e2ebd.el9.x86_6 13/34 2024-04-27T01:00:09.956 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2447.gb22e2ebd.el9.x86_6 13/34 2024-04-27T01:00:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-27T01:00:10.102 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-27T01:00:10.133 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-04-27T01:00:10.173 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-27T01:00:10.206 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-27T01:00:10.236 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-04-27T01:00:10.269 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-27T01:00:10.581 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-27T01:00:10.632 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 22/34 2024-04-27T01:00:10.721 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 22/34 2024-04-27T01:00:10.752 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 23/34 2024-04-27T01:00:10.788 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-27T01:00:11.257 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-04-27T01:00:11.283 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 26/34 2024-04-27T01:00:11.355 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 26/34 2024-04-27T01:00:11.433 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 27/34 2024-04-27T01:00:11.471 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x8 28/34 2024-04-27T01:00:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 29/34 2024-04-27T01:00:13.959 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 29/34 2024-04-27T01:00:14.504 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 29/34 2024-04-27T01:00:14.620 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.0.0-2447.gb22e2ebd.el9.x86_64 30/34 2024-04-27T01:00:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.0.0-2447.gb22e2ebd.el9.x86_64 30/34 2024-04-27T01:00:14.714 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-27T01:00:14.715 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-27T01:00:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.0.0-2447.gb22e2ebd.el9.x86_64 31/34 2024-04-27T01:00:15.241 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 873 kB/s | 19 MB 00:22 2024-04-27T01:00:24.348 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2447.gb22e2ebd.el9.x86_64 31/34 2024-04-27T01:00:24.349 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-04-27T01:00:24.349 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-04-27T01:00:24.349 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-04-27T01:00:24.349 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-04-27T01:00:24.349 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-04-27T01:00:24.349 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-04-27T01:00:24.349 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-04-27T01:00:24.349 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:25.311 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 32/34 2024-04-27T01:00:25.342 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 32/34 2024-04-27T01:00:25.342 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:00:25.342 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-27T01:00:25.342 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T01:00:25.342 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T01:00:25.342 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:25.344 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-27T01:00:25.430 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-27T01:00:25.431 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-27T01:00:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-27T01:00:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/34 2024-04-27T01:00:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2/34 2024-04-27T01:00:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 3/34 2024-04-27T01:00:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.0.0-2447.gb22e2ebd.el9.x86_64 4/34 2024-04-27T01:00:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 5/34 2024-04-27T01:00:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.0.0-2447.gb22e2ebd.el9.x86_6 6/34 2024-04-27T01:00:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 7/34 2024-04-27T01:00:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2447.gb22e2ebd.el9. 8/34 2024-04-27T01:00:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x8 9/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.0.0-2447.gb22e2ebd.el9.x86_64 10/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.0.0-2447.gb22e2ebd.el9.x86_64 11/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 12/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 13/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-27T01:00:26.731 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-27T01:00:26.732 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-04-27T01:00:26.732 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-04-27T01:00:26.732 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-27T01:00:26.732 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-27T01:00:26.732 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-04-27T01:00:26.732 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-27T01:00:26.732 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-27T01:00:26.733 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 31/34 2024-04-27T01:00:26.733 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-27T01:00:26.733 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 33/34 2024-04-27T01:00:26.957 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 774 kB/s | 6.2 MB 00:08 2024-04-27T01:00:27.493 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-27T01:00:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:27.494 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-04-27T01:00:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.494 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.494 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:00:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-27T01:00:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.495 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-27T01:00:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-27T01:00:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-27T01:00:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-27T01:00:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-27T01:00:27.496 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-27T01:00:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-27T01:00:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-27T01:00:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-27T01:00:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-27T01:00:27.497 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-27T01:00:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.498 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-27T01:00:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-27T01:00:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-27T01:00:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-3.el9.x86_64 2024-04-27T01:00:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-27T01:00:27.499 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:27.499 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:00:27.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test 2024-04-27T01:00:28.348 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:29 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:00:28.472 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 48 M 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 164 k 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:28.473 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-04-27T01:00:28.474 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:28.474 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 49 M 2024-04-27T01:00:28.474 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 207 M 2024-04-27T01:00:28.474 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:00:28.716 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-27T01:00:28.779 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 305 kB 00:00 2024-04-27T01:00:28.838 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-27T01:00:29.258 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): libcephsqlite-19.0.0-2447.gb22e2ebd.el9. 246 kB/s | 164 kB 00:00 2024-04-27T01:00:30.650 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-27T01:00:35.575 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-27T01:00:35.656 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): ceph-test-19.0.0-2447.gb22e2ebd.el9.x86_ 6.8 MB/s | 48 MB 00:07 2024-04-27T01:00:35.657 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:00:35.658 INFO:teuthology.orchestra.run.smithi162.stdout:Total 6.8 MB/s | 49 MB 00:07 2024-04-27T01:00:35.682 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:00:35.701 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:00:35.701 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:00:36.069 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:00:36.069 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:00:36.587 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:00:36.665 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-27T01:00:36.705 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-27T01:00:36.732 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.0.0-2447.gb22e2ebd.el9.x86_64 3/4 2024-04-27T01:00:37.726 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:00:37.728 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:00:37.728 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-27T01:00:37.728 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:00:37.728 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:00:37.728 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 10 M 2024-04-27T01:00:37.728 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-04-27T01:00:37.728 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 3.4 M 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 3.1 M 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 5.4 M 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 21 M 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 25 k 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 745 k 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-27T01:00:37.729 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 501 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 5.3 M 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 45 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 133 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 161 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 321 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 298 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 99 k 2024-04-27T01:00:37.730 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout:Install 30 Packages 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-04-27T01:00:37.731 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:00:37.732 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 60 M 2024-04-27T01:00:37.732 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:00:39.793 INFO:teuthology.orchestra.run.smithi181.stdout:(1/32): ceph-common-19.0.0-2447.gb22e2ebd.el9.x 12 MB/s | 21 MB 00:01 2024-04-27T01:00:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:(2/32): ceph-selinux-19.0.0-2447.gb22e2ebd.el9. 611 kB/s | 25 kB 00:00 2024-04-27T01:00:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:(3/32): libcephfs2-19.0.0-2447.gb22e2ebd.el9.x8 7.3 MB/s | 745 kB 00:00 2024-04-27T01:00:39.994 INFO:teuthology.orchestra.run.smithi181.stdout:(4/32): libradosstriper1-19.0.0-2447.gb22e2ebd. 8.5 MB/s | 501 kB 00:00 2024-04-27T01:00:40.378 INFO:teuthology.orchestra.run.smithi181.stdout:(5/32): librgw2-19.0.0-2447.gb22e2ebd.el9.x86_6 14 MB/s | 5.3 MB 00:00 2024-04-27T01:00:40.419 INFO:teuthology.orchestra.run.smithi181.stdout:(6/32): python3-ceph-argparse-19.0.0-2447.gb22e 1.1 MB/s | 45 kB 00:00 2024-04-27T01:00:40.470 INFO:teuthology.orchestra.run.smithi181.stdout:(7/32): python3-ceph-common-19.0.0-2447.gb22e2e 2.6 MB/s | 133 kB 00:00 2024-04-27T01:00:40.528 INFO:teuthology.orchestra.run.smithi181.stdout:(8/32): python3-cephfs-19.0.0-2447.gb22e2ebd.el 2.7 MB/s | 161 kB 00:00 2024-04-27T01:00:40.587 INFO:teuthology.orchestra.run.smithi181.stdout:(9/32): python3-rados-19.0.0-2447.gb22e2ebd.el9 5.4 MB/s | 321 kB 00:00 2024-04-27T01:00:40.695 INFO:teuthology.orchestra.run.smithi181.stdout:(10/32): ceph-radosgw-19.0.0-2447.gb22e2ebd.el9 3.8 MB/s | 10 MB 00:02 2024-04-27T01:00:40.721 INFO:teuthology.orchestra.run.smithi181.stdout:(11/32): python3-rbd-19.0.0-2447.gb22e2ebd.el9. 2.2 MB/s | 298 kB 00:00 2024-04-27T01:00:40.754 INFO:teuthology.orchestra.run.smithi181.stdout:(12/32): python3-rgw-19.0.0-2447.gb22e2ebd.el9. 1.7 MB/s | 99 kB 00:00 2024-04-27T01:00:40.879 INFO:teuthology.orchestra.run.smithi181.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 674 kB/s | 107 kB 00:00 2024-04-27T01:00:40.905 INFO:teuthology.orchestra.run.smithi181.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-04-27T01:00:40.930 INFO:teuthology.orchestra.run.smithi181.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-04-27T01:00:41.014 INFO:teuthology.orchestra.run.smithi181.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-04-27T01:00:41.064 INFO:teuthology.orchestra.run.smithi181.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-04-27T01:00:41.089 INFO:teuthology.orchestra.run.smithi181.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 562 kB/s | 42 kB 00:00 2024-04-27T01:00:41.114 INFO:teuthology.orchestra.run.smithi181.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-27T01:00:41.139 INFO:teuthology.orchestra.run.smithi181.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 716 kB/s | 36 kB 00:00 2024-04-27T01:00:41.165 INFO:teuthology.orchestra.run.smithi181.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-04-27T01:00:41.244 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2447.gb22e2ebd.el9.x86_64 3/4 2024-04-27T01:00:41.321 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.0.0-2447.gb22e2ebd.el9.x86_64 4/4 2024-04-27T01:00:41.349 INFO:teuthology.orchestra.run.smithi181.stdout:(22/32): ceph-base-19.0.0-2447.gb22e2ebd.el9.x8 1.6 MB/s | 5.4 MB 00:03 2024-04-27T01:00:41.465 INFO:teuthology.orchestra.run.smithi181.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-04-27T01:00:41.524 INFO:teuthology.orchestra.run.smithi181.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 803 kB/s | 309 kB 00:00 2024-04-27T01:00:41.550 INFO:teuthology.orchestra.run.smithi181.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 579 kB/s | 49 kB 00:00 2024-04-27T01:00:41.633 INFO:teuthology.orchestra.run.smithi181.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-04-27T01:00:41.784 INFO:teuthology.orchestra.run.smithi181.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.5 MB/s | 839 kB 00:00 2024-04-27T01:00:41.809 INFO:teuthology.orchestra.run.smithi181.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-27T01:00:41.901 INFO:teuthology.orchestra.run.smithi181.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-27T01:00:42.151 INFO:teuthology.orchestra.run.smithi181.stdout:(30/32): librbd1-19.0.0-2447.gb22e2ebd.el9.x86_ 13 MB/s | 3.1 MB 00:00 2024-04-27T01:00:42.518 INFO:teuthology.orchestra.run.smithi181.stdout:(31/32): libarrow-9.0.0-11.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-04-27T01:00:43.068 INFO:teuthology.orchestra.run.smithi181.stdout:(32/32): librados2-19.0.0-2447.gb22e2ebd.el9.x8 2.7 MB/s | 3.4 MB 00:01 2024-04-27T01:00:43.070 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:00:43.071 INFO:teuthology.orchestra.run.smithi181.stdout:Total 11 MB/s | 60 MB 00:05 2024-04-27T01:00:43.166 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:00:43.202 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:00:43.203 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:00:43.643 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-2:19.0.0-2447.gb22e2ebd.el9.x86_64 4/4 2024-04-27T01:00:43.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/4 2024-04-27T01:00:43.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2/4 2024-04-27T01:00:43.644 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-27T01:00:43.665 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:00:43.665 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:00:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-27T01:00:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:43.918 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:00:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-27T01:00:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-27T01:00:43.918 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:43.919 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:00:44.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph 2024-04-27T01:00:44.713 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:45 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:00:44.839 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:00:44.841 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:44.841 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 6.6 k 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 2.4 M 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 1.5 M 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 249 k 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 4.7 M 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 17 M 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-27T01:00:44.842 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-27T01:00:44.843 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-27T01:00:44.844 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-27T01:00:44.845 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-27T01:00:44.845 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:44.845 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:00:44.845 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:44.845 INFO:teuthology.orchestra.run.smithi162.stdout:Install 36 Packages 2024-04-27T01:00:44.845 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:44.845 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 30 M 2024-04-27T01:00:44.846 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 107 M 2024-04-27T01:00:44.846 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:00:45.139 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:00:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-27T01:00:45.264 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-27T01:00:45.290 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-27T01:00:45.313 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-27T01:00:45.336 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-27T01:00:45.487 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-27T01:00:45.550 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-27T01:00:45.878 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-27T01:00:45.986 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 7/34 2024-04-27T01:00:46.078 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 7/34 2024-04-27T01:00:46.170 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.0.0-2447.gb22e2ebd.el9.x86_64 8/34 2024-04-27T01:00:46.204 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 9/34 2024-04-27T01:00:46.248 INFO:teuthology.orchestra.run.smithi162.stdout:(1/36): ceph-19.0.0-2447.gb22e2ebd.el9.x86_64.r 30 kB/s | 6.6 kB 00:00 2024-04-27T01:00:46.266 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 9/34 2024-04-27T01:00:46.295 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-27T01:00:46.339 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.0.0-2447.gb22e2ebd.el9. 11/34 2024-04-27T01:00:46.409 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.0.0-2447.gb22e2ebd.el9.x86_64 12/34 2024-04-27T01:00:46.430 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.0.0-2447.gb22e2ebd.el9.x86_6 13/34 2024-04-27T01:00:46.495 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2447.gb22e2ebd.el9.x86_6 13/34 2024-04-27T01:00:46.529 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-27T01:00:46.639 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-27T01:00:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-04-27T01:00:46.681 INFO:teuthology.orchestra.run.smithi162.stdout:(2/36): ceph-mgr-19.0.0-2447.gb22e2ebd.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-04-27T01:00:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-27T01:00:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-27T01:00:46.766 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-04-27T01:00:46.807 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-27T01:00:47.123 INFO:teuthology.orchestra.run.smithi162.stdout:(3/36): ceph-mds-19.0.0-2447.gb22e2ebd.el9.x86_ 2.2 MB/s | 2.4 MB 00:01 2024-04-27T01:00:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-27T01:00:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 22/34 2024-04-27T01:00:47.232 INFO:teuthology.orchestra.run.smithi162.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2447.gb22e 2.2 MB/s | 249 kB 00:00 2024-04-27T01:00:47.260 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 22/34 2024-04-27T01:00:47.294 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 23/34 2024-04-27T01:00:47.335 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-27T01:00:47.449 INFO:teuthology.orchestra.run.smithi162.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-04-27T01:00:47.557 INFO:teuthology.orchestra.run.smithi162.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-04-27T01:00:47.616 INFO:teuthology.orchestra.run.smithi162.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-27T01:00:47.708 INFO:teuthology.orchestra.run.smithi162.stdout:(8/36): ceph-mon-19.0.0-2447.gb22e2ebd.el9.x86_ 3.2 MB/s | 4.7 MB 00:01 2024-04-27T01:00:47.749 INFO:teuthology.orchestra.run.smithi162.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.0 MB/s | 135 kB 00:00 2024-04-27T01:00:47.804 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-04-27T01:00:47.839 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 26/34 2024-04-27T01:00:47.841 INFO:teuthology.orchestra.run.smithi162.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.2 MB/s | 215 kB 00:00 2024-04-27T01:00:47.902 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 26/34 2024-04-27T01:00:47.908 INFO:teuthology.orchestra.run.smithi162.stdout:(11/36): python3-requests-2.25.1-8.el9.noarch.r 623 kB/s | 125 kB 00:00 2024-04-27T01:00:47.978 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 27/34 2024-04-27T01:00:47.983 INFO:teuthology.orchestra.run.smithi162.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-04-27T01:00:48.008 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x8 28/34 2024-04-27T01:00:48.025 INFO:teuthology.orchestra.run.smithi162.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 834 kB/s | 35 kB 00:00 2024-04-27T01:00:48.051 INFO:teuthology.orchestra.run.smithi162.stdout:(14/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-04-27T01:00:48.076 INFO:teuthology.orchestra.run.smithi162.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-27T01:00:48.101 INFO:teuthology.orchestra.run.smithi162.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-04-27T01:00:48.310 INFO:teuthology.orchestra.run.smithi162.stdout:(17/36): ceph-osd-19.0.0-2447.gb22e2ebd.el9.x86 10 MB/s | 17 MB 00:01 2024-04-27T01:00:48.393 INFO:teuthology.orchestra.run.smithi162.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 70 kB/s | 22 kB 00:00 2024-04-27T01:00:48.452 INFO:teuthology.orchestra.run.smithi162.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 307 kB/s | 43 kB 00:00 2024-04-27T01:00:48.502 INFO:teuthology.orchestra.run.smithi162.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 377 kB/s | 151 kB 00:00 2024-04-27T01:00:48.544 INFO:teuthology.orchestra.run.smithi162.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-04-27T01:00:48.603 INFO:teuthology.orchestra.run.smithi162.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 822 kB/s | 172 kB 00:00 2024-04-27T01:00:48.620 INFO:teuthology.orchestra.run.smithi162.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-04-27T01:00:48.645 INFO:teuthology.orchestra.run.smithi162.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-04-27T01:00:48.662 INFO:teuthology.orchestra.run.smithi162.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 394 kB/s | 23 kB 00:00 2024-04-27T01:00:48.687 INFO:teuthology.orchestra.run.smithi162.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2024-04-27T01:00:48.712 INFO:teuthology.orchestra.run.smithi162.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-04-27T01:00:48.738 INFO:teuthology.orchestra.run.smithi162.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-04-27T01:00:48.763 INFO:teuthology.orchestra.run.smithi162.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-04-27T01:00:48.788 INFO:teuthology.orchestra.run.smithi162.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-27T01:00:48.814 INFO:teuthology.orchestra.run.smithi162.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-04-27T01:00:48.839 INFO:teuthology.orchestra.run.smithi162.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-27T01:00:48.864 INFO:teuthology.orchestra.run.smithi162.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-04-27T01:00:48.914 INFO:teuthology.orchestra.run.smithi162.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 401 kB/s | 20 kB 00:00 2024-04-27T01:00:48.940 INFO:teuthology.orchestra.run.smithi162.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-04-27T01:00:48.973 INFO:teuthology.orchestra.run.smithi162.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-04-27T01:00:48.976 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:00:48.977 INFO:teuthology.orchestra.run.smithi162.stdout:Total 7.3 MB/s | 30 MB 00:04 2024-04-27T01:00:49.114 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:00:49.155 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:00:49.155 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:00:49.455 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:00:49.455 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:00:50.197 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 29/34 2024-04-27T01:00:50.237 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 29/34 2024-04-27T01:00:50.745 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:00:50.775 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 29/34 2024-04-27T01:00:50.800 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-27T01:00:50.815 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.0.0-2447.gb22e2ebd.el9.x86_64 30/34 2024-04-27T01:00:50.842 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-27T01:00:50.870 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-27T01:00:50.890 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.0.0-2447.gb22e2ebd.el9.x86_64 30/34 2024-04-27T01:00:50.890 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-27T01:00:50.890 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-27T01:00:50.890 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:00:50.925 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.0.0-2447.gb22e2ebd.el9.x86_64 31/34 2024-04-27T01:00:50.941 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-27T01:00:50.994 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-27T01:00:51.026 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-27T01:00:51.119 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-27T01:00:51.205 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-27T01:00:51.246 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-27T01:00:51.281 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-27T01:00:51.311 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-27T01:00:51.345 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-27T01:00:51.378 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-27T01:00:51.410 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-27T01:00:51.453 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-27T01:00:51.489 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-27T01:00:51.526 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-27T01:00:51.599 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-27T01:00:51.747 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-27T01:00:51.825 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-27T01:00:51.880 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-27T01:00:51.918 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-27T01:00:51.975 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-27T01:00:52.031 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-27T01:00:52.226 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-27T01:00:52.271 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-27T01:00:52.349 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-27T01:00:52.439 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-27T01:00:52.478 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-27T01:00:52.595 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-27T01:00:52.760 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2447.gb22e2ebd.el9. 31/36 2024-04-27T01:00:52.789 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.0.0-2447.gb22e2ebd.el9.x86_64 32/36 2024-04-27T01:00:52.814 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2447.gb22e2ebd.el9.x86_64 32/36 2024-04-27T01:00:52.814 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:00:52.814 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-27T01:00:52.814 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T01:00:52.814 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T01:00:52.814 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:54.248 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 33/36 2024-04-27T01:00:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 33/36 2024-04-27T01:00:54.274 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:00:54.274 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-27T01:00:54.274 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T01:00:54.274 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T01:00:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:54.715 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.0.0-2447.gb22e2ebd.el9.x86_64 34/36 2024-04-27T01:00:54.738 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.0.0-2447.gb22e2ebd.el9.x86_64 34/36 2024-04-27T01:00:54.739 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:00:54.739 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-27T01:00:54.739 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T01:00:54.739 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T01:00:54.739 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:54.957 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.0.0-2447.gb22e2ebd.el9.x86_64 35/36 2024-04-27T01:00:54.980 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.0.0-2447.gb22e2ebd.el9.x86_64 35/36 2024-04-27T01:00:54.980 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:00:54.980 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-27T01:00:54.980 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T01:00:54.981 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T01:00:54.981 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:55.036 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.0.0-2447.gb22e2ebd.el9.x86_64 36/36 2024-04-27T01:00:56.771 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-2:19.0.0-2447.gb22e2ebd.el9.x86_64 36/36 2024-04-27T01:00:56.771 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/36 2024-04-27T01:00:56.771 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2/36 2024-04-27T01:00:56.771 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.0.0-2447.gb22e2ebd.el9.x86_64 3/36 2024-04-27T01:00:56.771 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.0.0-2447.gb22e2ebd.el9.x86_64 4/36 2024-04-27T01:00:56.771 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 5/36 2024-04-27T01:00:56.771 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2447.gb22e2ebd.el9. 6/36 2024-04-27T01:00:56.771 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-27T01:00:56.771 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-27T01:00:56.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-27T01:00:56.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-27T01:00:56.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-27T01:00:56.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-27T01:00:56.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-27T01:00:56.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-27T01:00:56.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-27T01:00:56.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-27T01:00:56.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-27T01:00:56.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-27T01:00:56.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-27T01:00:56.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-27T01:00:56.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-27T01:00:56.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-27T01:00:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-27T01:00:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:57.649 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:00:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:00:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:00:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-04-27T01:00:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-27T01:00:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-27T01:00:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-27T01:00:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-27T01:00:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:57.652 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:00:57.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base 2024-04-27T01:00:58.501 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:59 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:00:58.580 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:00:58.622 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:00:58.623 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-04-27T01:00:58.623 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:00:58.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install cephadm 2024-04-27T01:00:59.211 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:00 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:00:59.330 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 762 k 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 762 k 2024-04-27T01:00:59.331 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 768 k 2024-04-27T01:00:59.332 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:00:59.614 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm-19.0.0-2447.gb22e2ebd.el9.noarch.rpm 2.6 MB/s | 762 kB 00:00 2024-04-27T01:00:59.615 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:00:59.615 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.6 MB/s | 762 kB 00:00 2024-04-27T01:00:59.615 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:00:59.621 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:00:59.621 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:00:59.626 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:00:59.627 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:00:59.697 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:01:00.105 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/1 2024-04-27T01:01:00.182 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/1 2024-04-27T01:01:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2447.gb22e2ebd.el9.x86_64 31/34 2024-04-27T01:01:00.304 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-04-27T01:01:00.304 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-04-27T01:01:00.304 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-04-27T01:01:00.304 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-04-27T01:01:00.305 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-04-27T01:01:00.305 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-04-27T01:01:00.305 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-04-27T01:01:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:00.518 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/1 2024-04-27T01:01:00.756 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/1 2024-04-27T01:01:00.756 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:00.757 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:01:00.757 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:01:00.757 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:00.757 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:01:00.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache 2024-04-27T01:01:01.267 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 32/34 2024-04-27T01:01:01.290 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 32/34 2024-04-27T01:01:01.290 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:01:01.290 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-27T01:01:01.290 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T01:01:01.290 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-27T01:01:01.290 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:01.292 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-27T01:01:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-27T01:01:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-27T01:01:01.491 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:02 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:01:01.612 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:01:01.613 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:01:01.613 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 145 k 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 145 k 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 436 k 2024-04-27T01:01:01.614 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:01:01.979 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-19.0.0-2447.gb22e2e 398 kB/s | 145 kB 00:00 2024-04-27T01:01:01.979 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:01:01.980 INFO:teuthology.orchestra.run.smithi162.stdout:Total 396 kB/s | 145 kB 00:00 2024-04-27T01:01:01.980 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:01:01.987 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:01:01.987 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:01:02.043 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:01:02.044 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:01:02.181 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:01:02.280 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2447.gb22e2ebd. 1/1 2024-04-27T01:01:02.304 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2447.gb22e2ebd. 1/1 2024-04-27T01:01:02.305 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:01:02.305 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-27T01:01:02.305 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:02.623 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-27T01:01:02.623 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/34 2024-04-27T01:01:02.623 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2/34 2024-04-27T01:01:02.623 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 3/34 2024-04-27T01:01:02.623 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.0.0-2447.gb22e2ebd.el9.x86_64 4/34 2024-04-27T01:01:02.623 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 5/34 2024-04-27T01:01:02.623 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.0.0-2447.gb22e2ebd.el9.x86_6 6/34 2024-04-27T01:01:02.623 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 7/34 2024-04-27T01:01:02.623 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2447.gb22e2ebd.el9. 8/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x8 9/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.0.0-2447.gb22e2ebd.el9.x86_64 10/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.0.0-2447.gb22e2ebd.el9.x86_64 11/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 12/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 13/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-27T01:01:02.624 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 31/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-27T01:01:02.625 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 33/34 2024-04-27T01:01:03.146 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2447.gb22e2ebd. 1/1 2024-04-27T01:01:03.146 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:01:03.146 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.146 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:03.146 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:01:03.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr 2024-04-27T01:01:03.401 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-27T01:01:03.401 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:03.401 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-04-27T01:01:03.401 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.401 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.401 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:01:03.401 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-27T01:01:03.401 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-27T01:01:03.402 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-27T01:01:03.403 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-3.el9.x86_64 2024-04-27T01:01:03.404 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-27T01:01:03.404 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:03.404 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:01:03.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-04-27T01:01:03.884 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:04 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:01:03.965 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-mgr-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:01:04.010 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:01:04.011 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-04-27T01:01:04.011 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:01:04.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-dashboard 2024-04-27T01:01:04.206 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:29 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:01:04.331 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:01:04.331 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:04.331 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-27T01:01:04.331 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:04.331 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:01:04.331 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 48 M 2024-04-27T01:01:04.331 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-27T01:01:04.331 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 164 k 2024-04-27T01:01:04.331 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-27T01:01:04.332 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-27T01:01:04.332 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:04.332 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:01:04.332 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:04.332 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-04-27T01:01:04.332 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:04.332 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 49 M 2024-04-27T01:01:04.332 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 207 M 2024-04-27T01:01:04.332 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:01:04.602 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:05 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:01:04.723 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:01:04.724 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:01:04.724 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-04-27T01:01:04.724 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:01:04.724 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:01:04.724 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 3.5 M 2024-04-27T01:01:04.724 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-04-27T01:01:04.724 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 32 k 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 15 k 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:01:04.725 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:01:04.726 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-04-27T01:01:04.726 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:04.726 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 8.6 M 2024-04-27T01:01:04.726 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 103 M 2024-04-27T01:01:04.726 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:01:05.484 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 305 kB 00:00 2024-04-27T01:01:05.559 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 854 kB/s | 64 kB 00:00 2024-04-27T01:01:05.945 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): libcephsqlite-19.0.0-2447.gb22e2ebd.el9. 262 kB/s | 164 kB 00:00 2024-04-27T01:01:09.620 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): ceph-test-19.0.0-2447.gb22e2ebd.el9.x86_ 11 MB/s | 48 MB 00:04 2024-04-27T01:01:09.622 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:01:09.622 INFO:teuthology.orchestra.run.smithi181.stdout:Total 9.2 MB/s | 49 MB 00:05 2024-04-27T01:01:09.646 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:01:09.664 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:01:09.664 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:01:10.028 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:01:10.028 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:01:10.546 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:01:10.630 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-27T01:01:10.669 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-27T01:01:10.697 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.0.0-2447.gb22e2ebd.el9.x86_64 3/4 2024-04-27T01:01:11.098 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2447.gb22 55 kB/s | 15 kB 00:00 2024-04-27T01:01:11.131 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2447.gb2 102 kB/s | 32 kB 00:00 2024-04-27T01:01:11.273 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-04-27T01:01:11.307 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-04-27T01:01:11.474 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): ceph-mgr-dashboard-19.0.0-2447.gb22e2eb 5.4 MB/s | 3.5 MB 00:00 2024-04-27T01:01:11.724 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-04-27T01:01:11.749 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-04-27T01:01:12.149 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 4.8 MB/s | 2.0 MB 00:00 2024-04-27T01:01:12.175 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 340 kB/s | 144 kB 00:00 2024-04-27T01:01:12.217 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 461 kB/s | 31 kB 00:00 2024-04-27T01:01:12.242 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-04-27T01:01:13.217 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 444 kB/s | 863 kB 00:01 2024-04-27T01:01:13.221 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:01:13.222 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.0 MB/s | 8.6 MB 00:08 2024-04-27T01:01:13.274 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:01:13.293 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:01:13.294 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:01:13.450 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:01:13.450 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:01:13.973 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:01:14.067 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-27T01:01:14.111 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-27T01:01:14.174 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-27T01:01:14.204 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-27T01:01:14.282 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-27T01:01:14.417 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-27T01:01:14.509 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-27T01:01:14.568 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-27T01:01:14.596 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-27T01:01:14.627 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2447.gb22e2ebd.el9 10/12 2024-04-27T01:01:15.201 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2447.gb22e2ebd.el9.x86_64 3/4 2024-04-27T01:01:15.277 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.0.0-2447.gb22e2ebd.el9.x86_64 4/4 2024-04-27T01:01:15.683 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2447.gb22e2ebd.el 11/12 2024-04-27T01:01:15.720 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2447.gb22e2ebd.el9.noa 12/12 2024-04-27T01:01:16.992 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2447.gb22e2ebd.el9.noa 12/12 2024-04-27T01:01:16.992 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2447.gb22e2ebd.el 1/12 2024-04-27T01:01:16.992 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2447.gb22e2ebd.el9.noa 2/12 2024-04-27T01:01:16.992 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2447.gb22e2ebd.el9 3/12 2024-04-27T01:01:16.992 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-27T01:01:16.992 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-27T01:01:16.992 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-27T01:01:16.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-27T01:01:16.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-27T01:01:16.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-27T01:01:16.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-27T01:01:16.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-27T01:01:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-27T01:01:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:17.296 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:01:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-27T01:01:17.296 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:17.297 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:01:17.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-27T01:01:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:19.0.0-2447.gb22e2ebd.el9.x86_64 4/4 2024-04-27T01:01:17.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/4 2024-04-27T01:01:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2/4 2024-04-27T01:01:17.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-27T01:01:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-27T01:01:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:17.854 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:01:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-27T01:01:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-27T01:01:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:17.855 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:01:18.053 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:19 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:01:18.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-04-27T01:01:18.174 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:01:18.175 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-04-27T01:01:18.175 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-04-27T01:01:18.175 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-04-27T01:01:18.175 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:01:18.175 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 7.4 M 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:18.176 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:01:18.177 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================= 2024-04-27T01:01:18.177 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-04-27T01:01:18.177 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:18.177 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 38 M 2024-04-27T01:01:18.177 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 209 M 2024-04-27T01:01:18.177 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:01:18.669 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:43 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:01:18.794 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:01:18.797 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:18.797 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-27T01:01:18.797 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:18.797 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:01:18.797 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 6.6 k 2024-04-27T01:01:18.797 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-27T01:01:18.797 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 2.4 M 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 1.5 M 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 249 k 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 4.7 M 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 17 M 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-27T01:01:18.798 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-27T01:01:18.799 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout:Install 36 Packages 2024-04-27T01:01:18.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:18.801 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 30 M 2024-04-27T01:01:18.801 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 107 M 2024-04-27T01:01:18.801 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:01:21.641 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 226 kB/s | 195 kB 00:00 2024-04-27T01:01:22.299 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 50 kB/s | 33 kB 00:00 2024-04-27T01:01:22.450 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 479 kB/s | 803 kB 00:01 2024-04-27T01:01:22.992 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): ceph-mgr-diskprediction-local-19.0.0-24 3.3 MB/s | 7.4 MB 00:02 2024-04-27T01:01:23.017 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 32 kB/s | 18 kB 00:00 2024-04-27T01:01:23.600 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 58 kB/s | 35 kB 00:00 2024-04-27T01:01:24.525 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 264 kB/s | 244 kB 00:00 2024-04-27T01:01:25.962 INFO:teuthology.orchestra.run.smithi181.stdout:(1/36): ceph-19.0.0-2447.gb22e2ebd.el9.x86_64.r 23 kB/s | 6.6 kB 00:00 2024-04-27T01:01:26.684 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 698 kB/s | 3.0 MB 00:04 2024-04-27T01:01:26.931 INFO:teuthology.orchestra.run.smithi181.stdout:(2/36): ceph-mgr-19.0.0-2447.gb22e2ebd.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-04-27T01:01:27.196 INFO:teuthology.orchestra.run.smithi181.stdout:(3/36): ceph-mds-19.0.0-2447.gb22e2ebd.el9.x86_ 1.6 MB/s | 2.4 MB 00:01 2024-04-27T01:01:27.296 INFO:teuthology.orchestra.run.smithi181.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2447.gb22e 2.4 MB/s | 249 kB 00:00 2024-04-27T01:01:27.446 INFO:teuthology.orchestra.run.smithi181.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-04-27T01:01:27.546 INFO:teuthology.orchestra.run.smithi181.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 12 MB/s | 1.2 MB 00:00 2024-04-27T01:01:27.567 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 284 kB/s | 250 kB 00:00 2024-04-27T01:01:27.596 INFO:teuthology.orchestra.run.smithi181.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-27T01:01:27.647 INFO:teuthology.orchestra.run.smithi181.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-04-27T01:01:27.722 INFO:teuthology.orchestra.run.smithi181.stdout:(9/36): ceph-mon-19.0.0-2447.gb22e2ebd.el9.x86_ 2.7 MB/s | 4.7 MB 00:01 2024-04-27T01:01:27.747 INFO:teuthology.orchestra.run.smithi181.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 1.2 MB/s | 125 kB 00:00 2024-04-27T01:01:27.881 INFO:teuthology.orchestra.run.smithi181.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-04-27T01:01:28.581 INFO:teuthology.orchestra.run.smithi181.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 226 kB/s | 188 kB 00:00 2024-04-27T01:01:28.673 INFO:teuthology.orchestra.run.smithi181.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 217 kB/s | 172 kB 00:00 2024-04-27T01:01:28.932 INFO:teuthology.orchestra.run.smithi181.stdout:(14/36): ceph-osd-19.0.0-2447.gb22e2ebd.el9.x86 8.4 MB/s | 17 MB 00:01 2024-04-27T01:01:29.417 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.0 MB/s | 5.1 MB 00:04 2024-04-27T01:01:29.465 INFO:teuthology.orchestra.run.smithi181.stdout:(15/36): python3-markupsafe-1.1.1-12.el9.x86_64 39 kB/s | 35 kB 00:00 2024-04-27T01:01:29.607 INFO:teuthology.orchestra.run.smithi181.stdout:(16/36): python3-pytz-2021.1-5.el9.noarch.rpm 55 kB/s | 51 kB 00:00 2024-04-27T01:01:29.666 INFO:teuthology.orchestra.run.smithi181.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 57 kB/s | 42 kB 00:00 2024-04-27T01:01:29.691 INFO:teuthology.orchestra.run.smithi181.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-04-27T01:01:29.717 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 724 kB/s | 4.7 MB 00:06 2024-04-27T01:01:30.008 INFO:teuthology.orchestra.run.smithi181.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 127 kB/s | 43 kB 00:00 2024-04-27T01:01:30.033 INFO:teuthology.orchestra.run.smithi181.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 356 kB/s | 151 kB 00:00 2024-04-27T01:01:30.117 INFO:teuthology.orchestra.run.smithi181.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 403 kB/s | 172 kB 00:00 2024-04-27T01:01:30.217 INFO:teuthology.orchestra.run.smithi181.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 58 kB/s | 11 kB 00:00 2024-04-27T01:01:30.400 INFO:teuthology.orchestra.run.smithi181.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 62 kB/s | 18 kB 00:00 2024-04-27T01:01:30.476 INFO:teuthology.orchestra.run.smithi181.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 89 kB/s | 23 kB 00:00 2024-04-27T01:01:30.518 INFO:teuthology.orchestra.run.smithi181.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 704 kB/s | 358 kB 00:00 2024-04-27T01:01:30.668 INFO:teuthology.orchestra.run.smithi181.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 73 kB/s | 19 kB 00:00 2024-04-27T01:01:30.743 INFO:teuthology.orchestra.run.smithi181.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 74 kB/s | 20 kB 00:00 2024-04-27T01:01:30.860 INFO:teuthology.orchestra.run.smithi181.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 135 kB/s | 46 kB 00:00 2024-04-27T01:01:31.010 INFO:teuthology.orchestra.run.smithi181.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 230 kB/s | 79 kB 00:00 2024-04-27T01:01:31.127 INFO:teuthology.orchestra.run.smithi181.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 61 kB/s | 16 kB 00:00 2024-04-27T01:01:31.244 INFO:teuthology.orchestra.run.smithi181.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 544 kB/s | 272 kB 00:00 2024-04-27T01:01:31.361 INFO:teuthology.orchestra.run.smithi181.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 257 kB/s | 90 kB 00:00 2024-04-27T01:01:31.395 INFO:teuthology.orchestra.run.smithi181.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 134 kB/s | 36 kB 00:00 2024-04-27T01:01:31.662 INFO:teuthology.orchestra.run.smithi181.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2024-04-27T01:01:31.753 INFO:teuthology.orchestra.run.smithi181.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 451 kB/s | 230 kB 00:00 2024-04-27T01:01:31.945 INFO:teuthology.orchestra.run.smithi181.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 731 kB/s | 427 kB 00:00 2024-04-27T01:01:31.950 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:01:31.950 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.3 MB/s | 30 MB 00:13 2024-04-27T01:01:32.086 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:01:32.127 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:01:32.127 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:01:32.420 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:01:32.420 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:01:33.756 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-27T01:01:33.797 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-27T01:01:33.834 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-27T01:01:33.906 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-27T01:01:33.949 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-27T01:01:33.982 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-27T01:01:34.078 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-27T01:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-27T01:01:34.276 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-27T01:01:34.311 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-27T01:01:34.341 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-27T01:01:34.375 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-27T01:01:34.409 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-27T01:01:34.449 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-27T01:01:34.484 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-27T01:01:34.519 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-27T01:01:34.557 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-27T01:01:34.630 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-27T01:01:34.778 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-27T01:01:34.865 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-27T01:01:34.918 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-27T01:01:34.956 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-27T01:01:35.005 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-27T01:01:35.060 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-27T01:01:35.257 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-27T01:01:35.302 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-27T01:01:35.381 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-27T01:01:35.494 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-27T01:01:35.543 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-27T01:01:35.660 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-27T01:01:35.815 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2447.gb22e2ebd.el9. 31/36 2024-04-27T01:01:35.844 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.0.0-2447.gb22e2ebd.el9.x86_64 32/36 2024-04-27T01:01:35.870 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2447.gb22e2ebd.el9.x86_64 32/36 2024-04-27T01:01:35.870 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:01:35.870 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-27T01:01:35.870 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T01:01:35.870 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-27T01:01:35.870 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:37.320 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 33/36 2024-04-27T01:01:37.346 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 33/36 2024-04-27T01:01:37.347 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:01:37.347 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-27T01:01:37.347 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T01:01:37.347 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-27T01:01:37.347 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:37.788 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.0.0-2447.gb22e2ebd.el9.x86_64 34/36 2024-04-27T01:01:37.810 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.0.0-2447.gb22e2ebd.el9.x86_64 34/36 2024-04-27T01:01:37.811 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:01:37.811 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-27T01:01:37.811 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T01:01:37.811 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-27T01:01:37.811 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:38.038 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.0.0-2447.gb22e2ebd.el9.x86_64 35/36 2024-04-27T01:01:38.061 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.0.0-2447.gb22e2ebd.el9.x86_64 35/36 2024-04-27T01:01:38.061 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:01:38.061 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-27T01:01:38.061 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T01:01:38.062 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-27T01:01:38.062 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:38.133 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.0.0-2447.gb22e2ebd.el9.x86_64 36/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:19.0.0-2447.gb22e2ebd.el9.x86_64 36/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.0.0-2447.gb22e2ebd.el9.x86_64 3/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.0.0-2447.gb22e2ebd.el9.x86_64 4/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 5/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2447.gb22e2ebd.el9. 6/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-27T01:01:39.877 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-27T01:01:39.878 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-27T01:01:39.879 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-27T01:01:39.879 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-27T01:01:39.879 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-27T01:01:39.879 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-27T01:01:39.879 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-27T01:01:39.879 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-27T01:01:39.879 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-27T01:01:39.879 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-27T01:01:39.880 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-27T01:01:39.880 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-27T01:01:39.880 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-27T01:01:39.880 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-27T01:01:40.707 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-27T01:01:40.707 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:40.707 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:01:40.707 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:40.707 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:40.707 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:40.707 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:01:40.707 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-27T01:01:40.708 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-27T01:01:40.709 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-27T01:01:40.710 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-27T01:01:40.710 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-27T01:01:40.710 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-27T01:01:40.710 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:40.710 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:01:41.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-04-27T01:01:41.721 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:06 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:01:41.800 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:01:41.842 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:01:41.843 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-27T01:01:41.843 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:01:41.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-04-27T01:01:42.428 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:07 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:01:42.548 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 762 k 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 762 k 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 768 k 2024-04-27T01:01:42.549 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:01:42.837 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-19.0.0-2447.gb22e2ebd.el9.noarch.rpm 2.6 MB/s | 762 kB 00:00 2024-04-27T01:01:42.837 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:01:42.838 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.6 MB/s | 762 kB 00:00 2024-04-27T01:01:42.838 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:01:42.843 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:01:42.843 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:01:42.849 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:01:42.849 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:01:42.922 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:01:43.384 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/1 2024-04-27T01:01:43.464 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/1 2024-04-27T01:01:43.803 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/1 2024-04-27T01:01:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/1 2024-04-27T01:01:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:44.084 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:01:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:01:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:44.085 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:01:44.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-04-27T01:01:44.834 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:09 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:01:44.954 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:01:44.955 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:44.955 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-04-27T01:01:44.955 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:44.955 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:01:44.955 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 145 k 2024-04-27T01:01:44.955 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:44.956 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:01:44.956 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:44.956 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-27T01:01:44.956 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:44.956 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 145 k 2024-04-27T01:01:44.956 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 436 k 2024-04-27T01:01:44.956 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:01:45.184 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-19.0.0-2447.gb22e2e 635 kB/s | 145 kB 00:00 2024-04-27T01:01:45.185 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:01:45.185 INFO:teuthology.orchestra.run.smithi181.stdout:Total 630 kB/s | 145 kB 00:00 2024-04-27T01:01:45.185 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:01:45.193 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:01:45.193 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:01:45.248 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:01:45.249 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:01:45.403 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:01:45.486 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2447.gb22e2ebd. 1/1 2024-04-27T01:01:45.511 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2447.gb22e2ebd. 1/1 2024-04-27T01:01:45.511 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:01:45.511 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-27T01:01:45.512 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:46.449 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2447.gb22e2ebd. 1/1 2024-04-27T01:01:46.449 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:46.449 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:01:46.449 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:01:46.449 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:46.449 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:01:46.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-04-27T01:01:47.179 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:12 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:01:47.259 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:01:47.302 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:01:47.303 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-27T01:01:47.303 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:01:47.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-04-27T01:01:47.887 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:12 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:01:48.007 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:01:48.008 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:48.008 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-27T01:01:48.008 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:48.008 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:01:48.008 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 3.5 M 2024-04-27T01:01:48.008 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-27T01:01:48.008 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-27T01:01:48.008 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 32 k 2024-04-27T01:01:48.008 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 15 k 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-04-27T01:01:48.009 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:01:48.010 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 8.6 M 2024-04-27T01:01:48.010 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 103 M 2024-04-27T01:01:48.010 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:01:49.338 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 764 kB/s | 16 MB 00:21 2024-04-27T01:01:49.340 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:01:49.340 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.2 MB/s | 38 MB 00:31 2024-04-27T01:01:49.692 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:01:49.720 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:01:49.720 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:01:50.128 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:01:50.129 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:01:50.964 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:01:51.017 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-27T01:01:51.053 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-27T01:01:51.154 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-27T01:01:51.187 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-27T01:01:51.365 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-27T01:01:51.394 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-27T01:01:51.470 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-27T01:01:52.126 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-27T01:01:52.249 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-27T01:01:53.486 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T01:01:54.535 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-27T01:01:54.576 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2447.gb22e2 12/12 2024-04-27T01:01:56.596 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2447.gb2 118 kB/s | 32 kB 00:00 2024-04-27T01:01:56.621 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2447.gb22 52 kB/s | 15 kB 00:00 2024-04-27T01:01:56.838 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-04-27T01:01:57.030 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-04-27T01:01:57.055 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-04-27T01:01:57.115 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2447.gb22e2 12/12 2024-04-27T01:01:57.115 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2447.gb22e2 1/12 2024-04-27T01:01:57.115 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-27T01:01:57.116 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-27T01:01:57.116 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-27T01:01:57.116 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-27T01:01:57.116 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-27T01:01:57.116 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-27T01:01:57.116 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-27T01:01:57.116 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-27T01:01:57.116 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T01:01:57.116 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-27T01:01:57.510 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-27T01:01:57.510 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-27T01:01:57.511 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:57.512 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:01:57.539 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 40 kB/s | 19 kB 00:00 2024-04-27T01:01:58.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-rook 2024-04-27T01:01:58.131 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 498 kB/s | 548 kB 00:01 2024-04-27T01:01:58.356 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 642 kB/s | 144 kB 00:00 2024-04-27T01:01:58.533 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:59 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:01:58.564 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 148 kB/s | 31 kB 00:00 2024-04-27T01:01:58.640 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): ceph-mgr-dashboard-19.0.0-2447.gb22e2e 1.5 MB/s | 3.5 MB 00:02 2024-04-27T01:01:58.653 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:01:58.655 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:01:58.655 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-04-27T01:01:58.655 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:01:58.655 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:01:58.655 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 49 k 2024-04-27T01:01:58.655 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-04-27T01:01:58.655 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-27T01:01:58.655 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-27T01:01:58.655 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout:Install 13 Packages 2024-04-27T01:01:58.656 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:01:58.657 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 2.2 M 2024-04-27T01:01:58.657 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 26 M 2024-04-27T01:01:58.657 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:01:58.757 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 982 kB/s | 188 kB 00:00 2024-04-27T01:01:59.082 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 1.3 MB/s | 2.0 MB 00:01 2024-04-27T01:01:59.084 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:01:59.084 INFO:teuthology.orchestra.run.smithi181.stdout:Total 797 kB/s | 8.6 MB 00:11 2024-04-27T01:01:59.137 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:01:59.155 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:01:59.155 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:01:59.309 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:01:59.309 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:01:59.833 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:01:59.916 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-27T01:01:59.959 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-27T01:02:00.006 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-27T01:02:00.044 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-27T01:02:00.121 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-27T01:02:00.255 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-27T01:02:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-27T01:02:00.383 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-27T01:02:00.411 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-27T01:02:00.443 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2447.gb22e2ebd.el9 10/12 2024-04-27T01:02:01.499 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2447.gb22e2ebd.el 11/12 2024-04-27T01:02:01.543 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2447.gb22e2ebd.el9.noa 12/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2447.gb22e2ebd.el9.noa 12/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2447.gb22e2ebd.el 1/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2447.gb22e2ebd.el9.noa 2/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2447.gb22e2ebd.el9 3/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-27T01:02:02.873 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-27T01:02:02.874 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-27T01:02:03.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-27T01:02:03.155 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-27T01:02:03.156 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-27T01:02:03.156 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-27T01:02:03.156 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:03.156 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:03.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-27T01:02:03.930 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:28 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:04.051 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:04.052 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-04-27T01:02:04.052 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 7.4 M 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-27T01:02:04.053 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-27T01:02:04.054 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-27T01:02:04.054 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-27T01:02:04.054 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-27T01:02:04.054 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-27T01:02:04.054 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:04.054 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:02:04.054 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-04-27T01:02:04.054 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-04-27T01:02:04.054 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:04.055 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 38 M 2024-04-27T01:02:04.055 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 209 M 2024-04-27T01:02:04.055 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:02:06.466 INFO:teuthology.orchestra.run.smithi162.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 303 kB/s | 26 kB 00:00 2024-04-27T01:02:06.482 INFO:teuthology.orchestra.run.smithi162.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 189 kB/s | 19 kB 00:00 2024-04-27T01:02:06.549 INFO:teuthology.orchestra.run.smithi162.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.4 MB/s | 159 kB 00:00 2024-04-27T01:02:06.574 INFO:teuthology.orchestra.run.smithi162.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-04-27T01:02:06.600 INFO:teuthology.orchestra.run.smithi162.stdout:(5/13): ceph-mgr-rook-19.0.0-2447.gb22e2ebd.el9 225 kB/s | 49 kB 00:00 2024-04-27T01:02:06.642 INFO:teuthology.orchestra.run.smithi162.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 804 kB/s | 54 kB 00:00 2024-04-27T01:02:06.667 INFO:teuthology.orchestra.run.smithi162.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.3 MB/s | 279 kB 00:00 2024-04-27T01:02:06.875 INFO:teuthology.orchestra.run.smithi162.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-04-27T01:02:06.909 INFO:teuthology.orchestra.run.smithi162.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-04-27T01:02:07.051 INFO:teuthology.orchestra.run.smithi162.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 418 kB/s | 59 kB 00:00 2024-04-27T01:02:07.118 INFO:teuthology.orchestra.run.smithi162.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 516 kB/s | 206 kB 00:00 2024-04-27T01:02:07.176 INFO:teuthology.orchestra.run.smithi162.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 718 kB/s | 90 kB 00:00 2024-04-27T01:02:07.227 INFO:teuthology.orchestra.run.smithi162.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.9 MB/s | 1.0 MB 00:00 2024-04-27T01:02:07.231 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:07.231 INFO:teuthology.orchestra.run.smithi162.stdout:Total 263 kB/s | 2.2 MB 00:08 2024-04-27T01:02:07.299 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:02:07.315 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:02:07.315 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:02:07.402 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:02:07.402 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:02:07.914 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:02:08.094 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-27T01:02:08.161 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-27T01:02:08.217 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-27T01:02:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-27T01:02:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-27T01:02:08.391 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-27T01:02:08.493 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-27T01:02:08.556 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-27T01:02:09.289 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-27T01:02:09.344 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-27T01:02:09.378 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-27T01:02:09.426 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-27T01:02:09.456 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.0.0-2447.gb22e2ebd.el9.noarch 13/13 2024-04-27T01:02:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2447.gb22e2ebd.el9.noarch 13/13 2024-04-27T01:02:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/13 2024-04-27T01:02:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-27T01:02:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-27T01:02:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-27T01:02:10.161 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-27T01:02:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-27T01:02:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-27T01:02:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-27T01:02:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-27T01:02:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-27T01:02:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-27T01:02:10.162 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-27T01:02:10.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-27T01:02:10.482 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:10.482 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:02:10.482 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:02:10.482 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-27T01:02:10.482 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:10.483 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:10.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-cephadm 2024-04-27T01:02:11.228 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:12 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:11.348 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:11.349 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:11.349 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-04-27T01:02:11.349 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:11.349 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:02:11.349 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 141 k 2024-04-27T01:02:11.349 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-04-27T01:02:11.349 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-27T01:02:11.350 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-27T01:02:11.350 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-27T01:02:11.350 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-27T01:02:11.350 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-27T01:02:11.350 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:11.350 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:02:11.350 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:11.350 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-04-27T01:02:11.350 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:11.351 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 7.0 M 2024-04-27T01:02:11.351 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 33 M 2024-04-27T01:02:11.351 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:02:18.246 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 598 kB/s | 195 kB 00:00 2024-04-27T01:02:18.288 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-04-27T01:02:18.380 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-04-27T01:02:18.430 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 366 kB/s | 18 kB 00:00 2024-04-27T01:02:18.514 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 422 kB/s | 35 kB 00:00 2024-04-27T01:02:19.331 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-24 5.3 MB/s | 7.4 MB 00:01 2024-04-27T01:02:19.531 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-04-27T01:02:19.623 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.2 MB/s | 3.0 MB 00:01 2024-04-27T01:02:19.740 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-04-27T01:02:20.115 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.0 MB/s | 4.7 MB 00:01 2024-04-27T01:02:21.240 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.0 MB/s | 5.1 MB 00:01 2024-04-27T01:02:21.809 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 909 kB/s | 248 kB 00:00 2024-04-27T01:02:21.959 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2447.gb22e2ebd.e 334 kB/s | 141 kB 00:00 2024-04-27T01:02:22.076 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.0 MB/s | 548 kB 00:00 2024-04-27T01:02:22.110 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 384 kB/s | 58 kB 00:00 2024-04-27T01:02:22.160 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 612 kB/s | 51 kB 00:00 2024-04-27T01:02:22.348 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.2 MB/s | 16 MB 00:02 2024-04-27T01:02:22.349 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:22.350 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.1 MB/s | 38 MB 00:18 2024-04-27T01:02:22.701 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:02:22.727 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:02:22.728 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:02:23.135 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:02:23.135 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:02:23.493 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.0 MB/s | 6.0 MB 00:01 2024-04-27T01:02:23.494 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:23.495 INFO:teuthology.orchestra.run.smithi162.stdout:Total 589 kB/s | 7.0 MB 00:12 2024-04-27T01:02:23.567 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:02:23.580 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:02:23.580 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:02:23.676 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:02:23.677 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:02:23.975 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:02:23.979 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:02:24.037 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-27T01:02:24.080 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-27T01:02:24.133 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-27T01:02:24.177 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-27T01:02:24.204 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-27T01:02:24.248 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-27T01:02:24.425 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-27T01:02:24.455 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-27T01:02:24.532 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-27T01:02:24.706 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-27T01:02:24.797 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-27T01:02:24.860 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-27T01:02:24.890 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 6/6 2024-04-27T01:02:25.172 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-27T01:02:25.293 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-27T01:02:25.698 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 6/6 2024-04-27T01:02:25.699 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/6 2024-04-27T01:02:25.699 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-27T01:02:25.699 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-27T01:02:25.699 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-27T01:02:25.699 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:25.961 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:26.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse 2024-04-27T01:02:26.505 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T01:02:26.718 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:27 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:26.838 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:26.839 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-04-27T01:02:26.839 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:26.839 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:02:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 889 k 2024-04-27T01:02:26.839 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-04-27T01:02:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-27T01:02:26.839 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:26.839 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:02:26.839 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:26.840 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-04-27T01:02:26.840 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:26.840 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 969 k 2024-04-27T01:02:26.840 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.8 M 2024-04-27T01:02:26.840 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:02:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-27T01:02:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2447.gb22e2 12/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2447.gb22e2 12/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2447.gb22e2 1/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-27T01:02:30.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-27T01:02:30.183 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-27T01:02:30.573 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 336 kB/s | 80 kB 00:00 2024-04-27T01:02:30.607 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-27T01:02:30.607 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:30.607 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-27T01:02:30.608 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:30.609 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:30.771 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-fuse-19.0.0-2447.gb22e2ebd.el9.x86_ 2.0 MB/s | 889 kB 00:00 2024-04-27T01:02:30.772 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:30.772 INFO:teuthology.orchestra.run.smithi162.stdout:Total 246 kB/s | 969 kB 00:03 2024-04-27T01:02:30.786 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:02:30.812 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:02:30.812 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:02:30.895 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:02:30.895 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:02:31.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-04-27T01:02:31.082 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:02:31.224 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-27T01:02:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2/2 2024-04-27T01:02:31.563 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:56 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:31.684 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:31.686 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:31.686 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-27T01:02:31.686 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:31.686 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:02:31.686 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 49 k 2024-04-27T01:02:31.686 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-27T01:02:31.686 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-27T01:02:31.686 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-27T01:02:31.686 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-27T01:02:31.686 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-04-27T01:02:31.687 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:31.688 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.2 M 2024-04-27T01:02:31.688 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-04-27T01:02:31.688 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:02:31.941 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2/2 2024-04-27T01:02:31.941 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/2 2024-04-27T01:02:32.194 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-27T01:02:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:32.195 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:02:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-27T01:02:32.195 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:32.195 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:32.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-volume 2024-04-27T01:02:32.991 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:33 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:33.115 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:33.116 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:33.116 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-04-27T01:02:33.116 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:33.116 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:02:33.116 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 264 k 2024-04-27T01:02:33.116 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-04-27T01:02:33.116 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-27T01:02:33.117 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:33.117 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:02:33.117 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:33.117 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-04-27T01:02:33.117 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:33.118 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 341 k 2024-04-27T01:02:33.118 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.5 M 2024-04-27T01:02:33.118 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): ceph-volume-19.0.0-2447.gb22e2ebd.el9.no 1.0 MB/s | 264 kB 00:00 2024-04-27T01:02:33.543 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 245 kB/s | 77 kB 00:00 2024-04-27T01:02:33.544 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:33.544 INFO:teuthology.orchestra.run.smithi162.stdout:Total 798 kB/s | 341 kB 00:00 2024-04-27T01:02:33.555 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:02:33.565 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:02:33.565 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:02:33.590 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:02:33.591 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:02:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:02:33.924 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-27T01:02:33.962 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.0.0-2447.gb22e2ebd.el9.noarch 2/2 2024-04-27T01:02:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.0.0-2447.gb22e2ebd.el9.noarch 2/2 2024-04-27T01:02:33.977 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:02:33.977 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-27T01:02:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:34.680 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/2 2024-04-27T01:02:34.950 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-27T01:02:34.950 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:34.950 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:02:34.950 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:02:34.950 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-27T01:02:34.951 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:34.951 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:35.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel 2024-04-27T01:02:35.804 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:36 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:35.930 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:35.931 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:35.931 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-04-27T01:02:35.931 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:35.931 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:02:35.931 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 127 k 2024-04-27T01:02:35.932 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:35.932 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:02:35.932 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:35.932 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-04-27T01:02:35.932 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:35.932 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 127 k 2024-04-27T01:02:35.932 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 456 k 2024-04-27T01:02:35.933 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:02:36.132 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-19.0.0-2447.gb22e2ebd.el9.x86_64 633 kB/s | 127 kB 00:00 2024-04-27T01:02:36.133 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:36.133 INFO:teuthology.orchestra.run.smithi162.stdout:Total 627 kB/s | 127 kB 00:00 2024-04-27T01:02:36.133 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:02:36.139 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:02:36.139 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:02:36.202 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:02:36.202 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:02:36.349 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:02:36.494 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:36.688 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 143 kB/s | 26 kB 00:00 2024-04-27T01:02:36.713 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 94 kB/s | 19 kB 00:00 2024-04-27T01:02:36.738 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): ceph-mgr-rook-19.0.0-2447.gb22e2ebd.el9 215 kB/s | 49 kB 00:00 2024-04-27T01:02:36.830 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-27T01:02:36.872 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-04-27T01:02:36.897 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 801 kB/s | 54 kB 00:00 2024-04-27T01:02:36.941 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:36.956 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-04-27T01:02:37.131 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-04-27T01:02:37.165 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-04-27T01:02:37.201 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:37.201 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:37.201 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:02:37.201 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:02:37.201 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:37.201 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:37.365 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-04-27T01:02:37.440 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 424 kB/s | 206 kB 00:00 2024-04-27T01:02:37.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2 2024-04-27T01:02:37.532 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 537 kB/s | 90 kB 00:00 2024-04-27T01:02:37.682 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-04-27T01:02:37.683 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:37.683 INFO:teuthology.orchestra.run.smithi181.stdout:Total 376 kB/s | 2.2 MB 00:05 2024-04-27T01:02:37.754 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:02:37.769 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:02:37.769 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:02:37.854 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:02:37.855 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:02:37.981 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:38 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:38.062 INFO:teuthology.orchestra.run.smithi162.stdout:Package libcephfs2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:38.106 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:38.107 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-04-27T01:02:38.107 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:38.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs-devel 2024-04-27T01:02:38.369 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:02:38.533 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-27T01:02:38.593 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-27T01:02:38.656 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-27T01:02:38.690 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:39 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:38.702 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-27T01:02:38.768 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-27T01:02:38.810 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 32 k 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 32 k 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 142 k 2024-04-27T01:02:38.811 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:02:38.870 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-27T01:02:38.980 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-27T01:02:39.028 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-27T01:02:39.060 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs-devel-19.0.0-2447.gb22e2ebd.el9.x86_6 128 kB/s | 32 kB 00:00 2024-04-27T01:02:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:Total 127 kB/s | 32 kB 00:00 2024-04-27T01:02:39.061 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:02:39.064 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:02:39.064 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:02:39.067 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:02:39.067 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:02:39.145 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:02:39.239 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:39.544 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:39.748 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:39.748 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:39.748 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:02:39.748 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:02:39.748 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:39.748 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:39.788 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-27T01:02:39.842 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-27T01:02:39.876 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-27T01:02:39.923 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-27T01:02:39.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2 2024-04-27T01:02:39.945 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.0.0-2447.gb22e2ebd.el9.noarch 13/13 2024-04-27T01:02:40.440 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:41 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:40.518 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:40.559 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:40.560 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-04-27T01:02:40.560 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:40.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1 2024-04-27T01:02:40.659 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2447.gb22e2ebd.el9.noarch 13/13 2024-04-27T01:02:40.659 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/13 2024-04-27T01:02:40.659 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-27T01:02:40.659 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-27T01:02:40.659 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-27T01:02:40.660 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-27T01:02:40.660 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-27T01:02:40.660 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-27T01:02:40.660 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-27T01:02:40.660 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-27T01:02:40.660 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-27T01:02:40.660 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-27T01:02:40.660 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-27T01:02:40.976 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-27T01:02:40.977 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-27T01:02:40.978 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-27T01:02:40.978 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-27T01:02:40.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:40.978 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:41.137 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:42 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:41.216 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:41.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-04-27T01:02:41.256 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:41.257 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-04-27T01:02:41.257 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:41.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados 2024-04-27T01:02:41.730 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:06 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:41.840 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:42 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:41.852 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 141 k 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-27T01:02:41.853 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:41.854 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:02:41.854 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:41.854 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-04-27T01:02:41.854 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:41.854 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 7.0 M 2024-04-27T01:02:41.854 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-04-27T01:02:41.854 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:02:41.921 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rados-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:41.963 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:41.964 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-04-27T01:02:41.964 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:42.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw 2024-04-27T01:02:42.552 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:43 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:42.632 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:42.673 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:42.674 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-04-27T01:02:42.674 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:42.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs 2024-04-27T01:02:43.251 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:44 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:43.330 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-cephfs-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:43.371 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:43.372 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-04-27T01:02:43.372 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:43.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd 2024-04-27T01:02:43.545 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): ceph-mgr-cephadm-19.0.0-2447.gb22e2ebd.e 608 kB/s | 141 kB 00:00 2024-04-27T01:02:43.570 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 963 kB/s | 248 kB 00:00 2024-04-27T01:02:43.929 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 161 kB/s | 58 kB 00:00 2024-04-27T01:02:43.954 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:44 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:44.036 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:44.071 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.9 MB/s | 6.0 MB 00:00 2024-04-27T01:02:44.077 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:44.078 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-04-27T01:02:44.078 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:44.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse 2024-04-27T01:02:44.196 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 842 kB/s | 548 kB 00:00 2024-04-27T01:02:44.338 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 125 kB/s | 51 kB 00:00 2024-04-27T01:02:44.339 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:44.339 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.8 MB/s | 7.0 MB 00:02 2024-04-27T01:02:44.413 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:02:44.424 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:02:44.424 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:02:44.520 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:02:44.520 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:02:44.661 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:45 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:44.780 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:44.781 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:44.781 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-04-27T01:02:44.781 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:44.781 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:02:44.781 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 85 k 2024-04-27T01:02:44.781 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:44.782 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:02:44.782 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:44.782 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-04-27T01:02:44.782 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:44.782 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 85 k 2024-04-27T01:02:44.782 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 231 k 2024-04-27T01:02:44.782 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:02:44.825 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:02:45.000 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-27T01:02:45.004 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-19.0.0-2447.gb22e2ebd.el9.x86_64.rpm 383 kB/s | 85 kB 00:00 2024-04-27T01:02:45.005 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:45.005 INFO:teuthology.orchestra.run.smithi162.stdout:Total 381 kB/s | 85 kB 00:00 2024-04-27T01:02:45.005 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:02:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:02:45.012 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:02:45.055 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-27T01:02:45.075 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:02:45.075 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:02:45.216 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:02:45.308 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:45.563 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-27T01:02:45.648 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:45.649 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-27T01:02:45.721 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-27T01:02:45.785 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 6/6 2024-04-27T01:02:45.874 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:45.874 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:45.874 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:02:45.874 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:02:45.874 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:45.874 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:46.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror 2024-04-27T01:02:46.516 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 6/6 2024-04-27T01:02:46.516 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/6 2024-04-27T01:02:46.516 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-27T01:02:46.517 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-27T01:02:46.517 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-27T01:02:46.517 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-27T01:02:46.635 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:47 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:46.756 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:46.757 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:46.757 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-04-27T01:02:46.757 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:46.757 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:02:46.757 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 3.1 M 2024-04-27T01:02:46.757 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:46.757 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:02:46.757 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:46.757 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-04-27T01:02:46.757 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:46.758 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.1 M 2024-04-27T01:02:46.758 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 13 M 2024-04-27T01:02:46.758 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:02:46.774 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-27T01:02:46.774 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:46.775 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:02:46.775 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:02:46.775 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-27T01:02:46.775 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-27T01:02:46.775 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-27T01:02:46.775 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-27T01:02:46.775 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-27T01:02:46.775 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:46.775 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:47.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-04-27T01:02:47.375 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-19.0.0-2447.gb22e2ebd.el9.x86_64.rpm 5.0 MB/s | 3.1 MB 00:00 2024-04-27T01:02:47.375 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:47.376 INFO:teuthology.orchestra.run.smithi162.stdout:Total 5.0 MB/s | 3.1 MB 00:00 2024-04-27T01:02:47.376 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:02:47.385 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:02:47.385 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:02:47.465 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:02:47.465 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:02:47.522 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:12 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 889 k 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-04-27T01:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 969 k 2024-04-27T01:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 2.8 M 2024-04-27T01:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:02:47.891 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:02:47.950 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:47.975 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:47.975 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:02:47.975 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-27T01:02:47.975 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T01:02:47.975 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T01:02:47.975 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:48.056 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 651 kB/s | 80 kB 00:00 2024-04-27T01:02:48.373 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-19.0.0-2447.gb22e2ebd.el9.x86_ 2.0 MB/s | 889 kB 00:00 2024-04-27T01:02:48.373 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:48.373 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.3 MB/s | 969 kB 00:00 2024-04-27T01:02:48.385 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:02:48.407 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:02:48.408 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:02:48.485 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:02:48.486 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:02:48.668 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:02:48.801 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-27T01:02:48.880 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2/2 2024-04-27T01:02:48.988 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:48.989 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:48.989 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:02:48.989 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:02:48.989 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:48.989 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:49.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd 2024-04-27T01:02:49.517 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2/2 2024-04-27T01:02:49.517 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/2 2024-04-27T01:02:49.747 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-27T01:02:49.747 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:49.747 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:02:49.747 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-27T01:02:49.747 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:49.747 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:49.748 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:50 ago on Sat 27 Apr 2024 12:59:59 AM UTC. 2024-04-27T01:02:49.874 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-04-27T01:02:49.874 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 170 k 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 170 k 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 494 k 2024-04-27T01:02:49.875 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-04-27T01:02:50.013 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-04-27T01:02:50.095 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-19.0.0-2447.gb22e2ebd.el9.x86_64.rpm 778 kB/s | 170 kB 00:00 2024-04-27T01:02:50.095 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:50.096 INFO:teuthology.orchestra.run.smithi162.stdout:Total 773 kB/s | 170 kB 00:00 2024-04-27T01:02:50.096 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-04-27T01:02:50.103 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-04-27T01:02:50.104 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-04-27T01:02:50.167 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-04-27T01:02:50.167 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-04-27T01:02:50.328 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-04-27T01:02:50.426 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:50.531 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:15 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:50.654 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.0.0-2447.gb22e2ebd.el9 ceph-noarch 264 k 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-04-27T01:02:50.655 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:50.656 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 341 k 2024-04-27T01:02:50.656 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.5 M 2024-04-27T01:02:50.656 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:02:50.812 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:50.978 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 500 kB/s | 77 kB 00:00 2024-04-27T01:02:51.055 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:51.055 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:51.055 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-04-27T01:02:51.055 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:02:51.055 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:02:51.055 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-04-27T01:02:51.061 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-volume-19.0.0-2447.gb22e2ebd.el9.no 1.1 MB/s | 264 kB 00:00 2024-04-27T01:02:51.062 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:51.062 INFO:teuthology.orchestra.run.smithi181.stdout:Total 838 kB/s | 341 kB 00:00 2024-04-27T01:02:51.073 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:02:51.083 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:02:51.083 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:02:51.108 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:02:51.108 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:02:51.244 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:02:51.347 DEBUG:teuthology.parallel:result is None 2024-04-27T01:02:51.411 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-27T01:02:51.447 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.0.0-2447.gb22e2ebd.el9.noarch 2/2 2024-04-27T01:02:51.463 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.0.0-2447.gb22e2ebd.el9.noarch 2/2 2024-04-27T01:02:51.463 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:02:51.463 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-27T01:02:51.463 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:52.134 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.0.0-2447.gb22e2ebd.el9.noarch 1/2 2024-04-27T01:02:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-27T01:02:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:52.416 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:02:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.0.0-2447.gb22e2ebd.el9.noarch 2024-04-27T01:02:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-27T01:02:52.416 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:52.416 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:52.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-04-27T01:02:53.297 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:18 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:53.419 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:53.420 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-04-27T01:02:53.420 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:53.420 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:02:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 127 k 2024-04-27T01:02:53.420 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:53.420 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:02:53.420 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:53.420 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-27T01:02:53.421 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:53.421 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 127 k 2024-04-27T01:02:53.421 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 456 k 2024-04-27T01:02:53.421 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-19.0.0-2447.gb22e2ebd.el9.x86_64 530 kB/s | 127 kB 00:00 2024-04-27T01:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout:Total 527 kB/s | 127 kB 00:00 2024-04-27T01:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:02:53.730 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:02:53.730 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:02:53.877 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:02:53.964 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:54.394 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:54.632 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:54.633 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:54.633 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:02:54.633 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:02:54.633 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:54.633 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:54.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-04-27T01:02:55.400 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:20 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:55.481 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:55.522 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:55.523 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-27T01:02:55.524 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:55.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-04-27T01:02:56.101 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:21 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:56.220 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:56.221 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:56.221 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-04-27T01:02:56.221 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:56.222 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:02:56.222 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 32 k 2024-04-27T01:02:56.222 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:56.222 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:02:56.222 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:02:56.222 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-27T01:02:56.222 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:56.222 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 32 k 2024-04-27T01:02:56.222 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 142 k 2024-04-27T01:02:56.222 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:02:56.397 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-19.0.0-2447.gb22e2ebd.el9.x86_6 182 kB/s | 32 kB 00:00 2024-04-27T01:02:56.397 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:02:56.397 INFO:teuthology.orchestra.run.smithi181.stdout:Total 181 kB/s | 32 kB 00:00 2024-04-27T01:02:56.397 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:02:56.401 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:02:56.401 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:02:56.403 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:02:56.403 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:02:56.491 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:02:56.609 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:56.942 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:57.171 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:02:57.171 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:57.171 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:02:57.172 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:02:57.172 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:02:57.172 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:57.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-04-27T01:02:57.857 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:22 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:57.935 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:57.976 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:57.977 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-27T01:02:57.977 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:58.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-04-27T01:02:58.552 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:23 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:58.635 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:58.678 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:58.679 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-27T01:02:58.679 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:58.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-04-27T01:02:59.293 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:24 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:02:59.373 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:02:59.414 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:02:59.415 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-27T01:02:59.415 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:02:59.482 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-04-27T01:03:00.006 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:25 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:03:00.086 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:03:00.127 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:03:00.128 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-27T01:03:00.128 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:03:00.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-04-27T01:03:00.702 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:25 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:03:00.781 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:03:00.823 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:03:00.824 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-27T01:03:00.824 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:03:00.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-04-27T01:03:01.389 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:26 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:03:01.468 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 is already installed. 2024-04-27T01:03:01.508 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:03:01.509 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-04-27T01:03:01.509 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:03:01.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-04-27T01:03:02.082 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:27 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:03:02.201 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:03:02.201 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:03:02.201 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 85 k 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 85 k 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 231 k 2024-04-27T01:03:02.202 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:03:02.417 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-19.0.0-2447.gb22e2ebd.el9.x86_64.rpm 398 kB/s | 85 kB 00:00 2024-04-27T01:03:02.417 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:03:02.418 INFO:teuthology.orchestra.run.smithi181.stdout:Total 395 kB/s | 85 kB 00:00 2024-04-27T01:03:02.418 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:03:02.424 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:03:02.425 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:03:02.487 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:03:02.487 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:03:02.629 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:03:02.722 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:03:03.090 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:03:03.321 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:03:03.321 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:03.321 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:03:03.321 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:03:03.321 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:03.321 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:03:03.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-04-27T01:03:04.081 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:29 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:03:04.205 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 3.1 M 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:04.206 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.1 M 2024-04-27T01:03:04.207 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 13 M 2024-04-27T01:03:04.207 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:03:04.854 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-19.0.0-2447.gb22e2ebd.el9.x86_64.rpm 4.8 MB/s | 3.1 MB 00:00 2024-04-27T01:03:04.855 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:03:04.855 INFO:teuthology.orchestra.run.smithi181.stdout:Total 4.8 MB/s | 3.1 MB 00:00 2024-04-27T01:03:04.856 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:03:04.865 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:03:04.865 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:03:04.946 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:03:04.947 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:03:05.371 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:03:05.430 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:03:05.454 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:03:05.455 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-27T01:03:05.455 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-27T01:03:05.455 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T01:03:05.455 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-27T01:03:05.455 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:06.503 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:03:06.503 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:06.503 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:03:06.504 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:03:06.504 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:06.504 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:03:06.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-04-27T01:03:07.264 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:32 ago on Sat 27 Apr 2024 01:00:35 AM UTC. 2024-04-27T01:03:07.390 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-04-27T01:03:07.391 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:03:07.391 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-04-27T01:03:07.391 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:03:07.391 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-04-27T01:03:07.391 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.0.0-2447.gb22e2ebd.el9 ceph 170 k 2024-04-27T01:03:07.391 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:07.391 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-04-27T01:03:07.391 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-04-27T01:03:07.392 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-04-27T01:03:07.392 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:07.392 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 170 k 2024-04-27T01:03:07.392 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 494 k 2024-04-27T01:03:07.392 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-04-27T01:03:07.699 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-19.0.0-2447.gb22e2ebd.el9.x86_64.rpm 555 kB/s | 170 kB 00:00 2024-04-27T01:03:07.700 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:03:07.700 INFO:teuthology.orchestra.run.smithi181.stdout:Total 552 kB/s | 170 kB 00:00 2024-04-27T01:03:07.700 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-04-27T01:03:07.708 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-04-27T01:03:07.708 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-04-27T01:03:07.771 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-04-27T01:03:07.771 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-04-27T01:03:07.919 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-04-27T01:03:08.029 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:03:08.432 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:03:08.656 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 1/1 2024-04-27T01:03:08.656 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:08.657 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-04-27T01:03:08.657 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.0.0-2447.gb22e2ebd.el9.x86_64 2024-04-27T01:03:08.657 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:03:08.657 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-04-27T01:03:08.893 DEBUG:teuthology.parallel:result is None 2024-04-27T01:03:08.893 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:03:09.118 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-27T01:03:09.143 INFO:teuthology.orchestra.run.smithi162.stdout:19.0.0-2447.gb22e2ebd.el9 2024-04-27T01:03:09.143 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2447.gb22e2ebd.el9 2024-04-27T01:03:09.144 INFO:teuthology.task.install:The correct ceph version 19.0.0-2447.gb22e2ebd is installed. 2024-04-27T01:03:09.146 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:03:09.385 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-27T01:03:09.405 INFO:teuthology.orchestra.run.smithi181.stdout:19.0.0-2447.gb22e2ebd.el9 2024-04-27T01:03:09.405 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2447.gb22e2ebd.el9 2024-04-27T01:03:09.406 INFO:teuthology.task.install:The correct ceph version 19.0.0-2447.gb22e2ebd is installed. 2024-04-27T01:03:09.407 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-27T01:03:09.408 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:03:09.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-27T01:03:09.437 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:03:09.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-27T01:03:09.469 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-27T01:03:09.469 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:03:09.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-04-27T01:03:09.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-27T01:03:09.569 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:03:09.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-04-27T01:03:09.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-27T01:03:09.669 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-27T01:03:09.670 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:03:09.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-27T01:03:09.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-27T01:03:09.757 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:03:09.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-27T01:03:09.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-27T01:03:09.845 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-27T01:03:09.846 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:03:09.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-04-27T01:03:09.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-27T01:03:09.934 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:03:09.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-04-27T01:03:09.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-27T01:03:10.020 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-27T01:03:10.118 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 3, 'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'cephadm_mode': 'cephadm-package'} 2024-04-27T01:03:10.119 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:03:10.119 INFO:tasks.cephadm:Cluster fsid is ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:03:10.119 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-27T01:03:10.119 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.162:6789]', 'mon.c': '[v1:172.21.15.162:6790]', 'mon.b': '[v1:172.21.15.181:6789]'} 2024-04-27T01:03:10.119 INFO:tasks.cephadm:First mon is mon.a on smithi162 2024-04-27T01:03:10.119 INFO:tasks.cephadm:First mgr is y 2024-04-27T01:03:10.119 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-27T01:03:10.120 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-04-27T01:03:10.143 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname $(hostname -s) 2024-04-27T01:03:10.177 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a on all hosts... 2024-04-27T01:03:10.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a pull 2024-04-27T01:03:10.185 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a pull 2024-04-27T01:03:11.830 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a... 2024-04-27T01:03:11.863 INFO:teuthology.orchestra.run.smithi181.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a... 2024-04-27T01:04:09.989 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-27T01:04:09.989 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_version": "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)", 2024-04-27T01:04:09.989 INFO:teuthology.orchestra.run.smithi181.stdout: "image_id": "87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae", 2024-04-27T01:04:09.990 INFO:teuthology.orchestra.run.smithi181.stdout: "repo_digests": [ 2024-04-27T01:04:09.990 INFO:teuthology.orchestra.run.smithi181.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6" 2024-04-27T01:04:09.990 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-27T01:04:09.990 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-27T01:04:10.164 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-04-27T01:04:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version": "ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev)", 2024-04-27T01:04:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "image_id": "87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae", 2024-04-27T01:04:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "repo_digests": [ 2024-04-27T01:04:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6" 2024-04-27T01:04:10.165 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-04-27T01:04:10.165 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-04-27T01:04:10.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-04-27T01:04:10.243 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph 2024-04-27T01:04:10.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-04-27T01:04:10.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /etc/ceph 2024-04-27T01:04:10.335 INFO:tasks.cephadm:Writing seed config... 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [global] ms type = async 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-04-27T01:04:10.337 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-27T01:04:10.338 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:04:10.338 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-27T01:04:10.353 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ea378f3c-0431-11ef-bc93-c7b262605968 mon election default strategy = 3 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-04-27T01:04:10.353 DEBUG:teuthology.orchestra.run.smithi162:mon.a> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.a.service 2024-04-27T01:04:10.395 DEBUG:teuthology.orchestra.run.smithi162:mgr.y> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mgr.y.service 2024-04-27T01:04:10.438 INFO:tasks.cephadm:Bootstrapping... 2024-04-27T01:04:10.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a -v bootstrap --fsid ea378f3c-0431-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.162:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:04:10.627 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-04-27T01:04:10.627 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a', '-v', 'bootstrap', '--fsid', 'ea378f3c-0431-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.162:6789]', '--skip-admin-label'] 2024-04-27T01:04:10.627 INFO:teuthology.orchestra.run.smithi162.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-27T01:04:10.627 INFO:teuthology.orchestra.run.smithi162.stdout:Verifying podman|docker is present... 2024-04-27T01:04:10.651 INFO:teuthology.orchestra.run.smithi162.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-27T01:04:10.652 INFO:teuthology.orchestra.run.smithi162.stdout:Verifying lvm2 is present... 2024-04-27T01:04:10.652 INFO:teuthology.orchestra.run.smithi162.stdout:Verifying time synchronization is in place... 2024-04-27T01:04:10.659 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-27T01:04:10.659 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-27T01:04:10.665 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-27T01:04:10.665 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout inactive 2024-04-27T01:04:10.671 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout enabled 2024-04-27T01:04:10.677 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout active 2024-04-27T01:04:10.677 INFO:teuthology.orchestra.run.smithi162.stdout:Unit chronyd.service is enabled and running 2024-04-27T01:04:10.678 INFO:teuthology.orchestra.run.smithi162.stdout:Repeating the final host check... 2024-04-27T01:04:10.701 INFO:teuthology.orchestra.run.smithi162.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-27T01:04:10.701 INFO:teuthology.orchestra.run.smithi162.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-27T01:04:10.701 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl is present 2024-04-27T01:04:10.702 INFO:teuthology.orchestra.run.smithi162.stdout:lvcreate is present 2024-04-27T01:04:10.708 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-27T01:04:10.708 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-27T01:04:10.714 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-27T01:04:10.714 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout inactive 2024-04-27T01:04:10.720 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout enabled 2024-04-27T01:04:10.726 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout active 2024-04-27T01:04:10.726 INFO:teuthology.orchestra.run.smithi162.stdout:Unit chronyd.service is enabled and running 2024-04-27T01:04:10.726 INFO:teuthology.orchestra.run.smithi162.stdout:Host looks OK 2024-04-27T01:04:10.726 INFO:teuthology.orchestra.run.smithi162.stdout:Cluster fsid: ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:04:10.726 INFO:teuthology.orchestra.run.smithi162.stdout:Acquiring lock 139800288738896 on /run/cephadm/ea378f3c-0431-11ef-bc93-c7b262605968.lock 2024-04-27T01:04:10.726 INFO:teuthology.orchestra.run.smithi162.stdout:Lock 139800288738896 acquired on /run/cephadm/ea378f3c-0431-11ef-bc93-c7b262605968.lock 2024-04-27T01:04:10.727 INFO:teuthology.orchestra.run.smithi162.stdout:Verifying IP 172.21.15.162 port 6789 ... 2024-04-27T01:04:10.727 INFO:teuthology.orchestra.run.smithi162.stdout:Base mon IP(s) is [172.21.15.162:6789], mon addrv is [v1:172.21.15.162:6789] 2024-04-27T01:04:10.730 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.162 metric 100 2024-04-27T01:04:10.730 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.162 metric 100 2024-04-27T01:04:10.733 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-27T01:04:10.733 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-27T01:04:10.733 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-27T01:04:10.735 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-27T01:04:10.735 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-27T01:04:10.735 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-27T01:04:10.735 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-27T01:04:10.735 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout inet6 fe80::7daa:2db1:c4b9:3a1c/64 scope link noprefixroute 2024-04-27T01:04:10.735 INFO:teuthology.orchestra.run.smithi162.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-27T01:04:10.736 INFO:teuthology.orchestra.run.smithi162.stdout:Mon IP `172.21.15.162` is in CIDR network `172.21.0.0/20` 2024-04-27T01:04:10.736 INFO:teuthology.orchestra.run.smithi162.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-04-27T01:04:10.736 INFO:teuthology.orchestra.run.smithi162.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-27T01:04:10.737 INFO:teuthology.orchestra.run.smithi162.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a... 2024-04-27T01:04:11.200 INFO:teuthology.orchestra.run.smithi162.stdout:/bin/podman: stdout 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae 2024-04-27T01:04:11.200 INFO:teuthology.orchestra.run.smithi162.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a... 2024-04-27T01:04:11.200 INFO:teuthology.orchestra.run.smithi162.stdout:/bin/podman: stderr Getting image source signatures 2024-04-27T01:04:11.200 INFO:teuthology.orchestra.run.smithi162.stdout:/bin/podman: stderr Copying blob sha256:8cad468f23ab438b6f898c52382f29d2bf4f5542ae8dbbba19c3cda86356dd4f 2024-04-27T01:04:11.200 INFO:teuthology.orchestra.run.smithi162.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-27T01:04:11.200 INFO:teuthology.orchestra.run.smithi162.stdout:/bin/podman: stderr Copying config sha256:87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae 2024-04-27T01:04:11.200 INFO:teuthology.orchestra.run.smithi162.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-27T01:04:12.182 INFO:teuthology.orchestra.run.smithi162.stdout:ceph: stdout ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev) 2024-04-27T01:04:12.182 INFO:teuthology.orchestra.run.smithi162.stdout:Ceph version: ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev) 2024-04-27T01:04:12.182 INFO:teuthology.orchestra.run.smithi162.stdout:Extracting ceph user uid/gid from container image... 2024-04-27T01:04:13.156 INFO:teuthology.orchestra.run.smithi162.stdout:stat: stdout 167 167 2024-04-27T01:04:13.156 INFO:teuthology.orchestra.run.smithi162.stdout:Creating initial keys... 2024-04-27T01:04:14.014 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph-authtool: stdout AQANTyxmbNajHhAA56e/FnQBdHmnom9lLZ07Dw== 2024-04-27T01:04:14.913 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph-authtool: stdout AQAOTyxmwwvdFhAAgvunUZOTWeZrElTLu6OfRg== 2024-04-27T01:04:15.770 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph-authtool: stdout AQAPTyxmdPIfDhAAWx3UBZYDq2Oo+K2njCvIXw== 2024-04-27T01:04:15.771 INFO:teuthology.orchestra.run.smithi162.stdout:Creating initial monmap... 2024-04-27T01:04:16.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-27T01:04:16.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-27T01:04:16.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:04:16.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-27T01:04:16.719 INFO:teuthology.orchestra.run.smithi162.stdout:monmaptool for a [v1:172.21.15.162:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-27T01:04:16.719 INFO:teuthology.orchestra.run.smithi162.stdout:setting min_mon_release = quincy 2024-04-27T01:04:16.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: set fsid to ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:04:16.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-27T01:04:16.720 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:04:16.720 INFO:teuthology.orchestra.run.smithi162.stdout:Creating mon... 2024-04-27T01:04:17.624 INFO:teuthology.orchestra.run.smithi162.stdout:create mon.a on 2024-04-27T01:04:17.839 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-27T01:04:17.978 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-27T01:04:18.154 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ea378f3c-0431-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-ea378f3c-0431-11ef-bc93-c7b262605968.target. 2024-04-27T01:04:18.154 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ea378f3c-0431-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-ea378f3c-0431-11ef-bc93-c7b262605968.target. 2024-04-27T01:04:18.398 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.a 2024-04-27T01:04:18.398 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-27T01:04:18.575 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ea378f3c-0431-11ef-bc93-c7b262605968.target.wants/ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-ea378f3c-0431-11ef-bc93-c7b262605968@.service. 2024-04-27T01:04:19.100 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T01:04:19.100 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout disabled 2024-04-27T01:04:19.107 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T01:04:19.108 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout inactive 2024-04-27T01:04:19.108 INFO:teuthology.orchestra.run.smithi162.stdout:firewalld.service is not enabled 2024-04-27T01:04:19.108 INFO:teuthology.orchestra.run.smithi162.stdout:Not possible to enable service . firewalld.service is not available 2024-04-27T01:04:19.108 INFO:teuthology.orchestra.run.smithi162.stdout:Waiting for mon to start... 2024-04-27T01:04:19.108 INFO:teuthology.orchestra.run.smithi162.stdout:Waiting for mon... 2024-04-27T01:04:20.175 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout cluster: 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout id: ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout services: 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.563627s) 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout data: 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout pgs: 2024-04-27T01:04:20.176 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.542+0000 7f17e9483700 1 Processor -- start 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.542+0000 7f17e9483700 1 -- start start 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.542+0000 7f17e9483700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f17e41093d0 con 0x7f17e4104fa0 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.542+0000 7f17e2ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f17e4104fa0 0x7f17e41053c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:44472/0 (socket says 172.21.15.162:44472) 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.542+0000 7f17e2ffd700 1 -- 172.21.15.162:0/4265712071 learned_addr learned my addr 172.21.15.162:0/4265712071 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.543+0000 7f17e1ffb700 1 -- 172.21.15.162:0/4265712071 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2997901193 0 0) 0x7f17e41093d0 con 0x7f17e4104fa0 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.543+0000 7f17e1ffb700 1 -- 172.21.15.162:0/4265712071 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17cc0035b0 con 0x7f17e4104fa0 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.543+0000 7f17e1ffb700 1 -- 172.21.15.162:0/4265712071 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 444262723 0 0) 0x7f17cc0035b0 con 0x7f17e4104fa0 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.543+0000 7f17e1ffb700 1 -- 172.21.15.162:0/4265712071 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e410a5b0 con 0x7f17e4104fa0 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.543+0000 7f17e1ffb700 1 -- 172.21.15.162:0/4265712071 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f17d4001580 con 0x7f17e4104fa0 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.543+0000 7f17e1ffb700 1 -- 172.21.15.162:0/4265712071 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17d40018b0 con 0x7f17e4104fa0 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.543+0000 7f17e1ffb700 1 -- 172.21.15.162:0/4265712071 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f17d4001c30 con 0x7f17e4104fa0 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.544+0000 7f17e9483700 1 -- 172.21.15.162:0/4265712071 >> v1:172.21.15.162:6789/0 conn(0x7f17e4104fa0 legacy=0x7f17e41053c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:20.177 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.544+0000 7f17e9483700 1 -- 172.21.15.162:0/4265712071 shutdown_connections 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.544+0000 7f17e9483700 1 -- 172.21.15.162:0/4265712071 >> 172.21.15.162:0/4265712071 conn(0x7f17e4100520 msgr2=0x7f17e4102980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.544+0000 7f17e9483700 1 -- 172.21.15.162:0/4265712071 shutdown_connections 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.544+0000 7f17e9483700 1 -- 172.21.15.162:0/4265712071 wait complete. 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.544+0000 7f17e9483700 1 Processor -- start 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17e9483700 1 -- start start 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17e9483700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f17e4199ae0 con 0x7f17e4104fa0 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17e2ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f17e4104fa0 0x7f17e41993d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:44482/0 (socket says 172.21.15.162:44482) 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17e2ffd700 1 -- 172.21.15.162:0/1619964771 learned_addr learned my addr 172.21.15.162:0/1619964771 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 513406651 0 0) 0x7f17e4199ae0 con 0x7f17e4104fa0 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17cc0031f0 con 0x7f17e4104fa0 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 716587445 0 0) 0x7f17cc0031f0 con 0x7f17e4104fa0 2024-04-27T01:04:20.178 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17e4199ae0 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f17d4001480 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4214817713 0 0) 0x7f17e4199ae0 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e4199cb0 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17e9483700 1 -- 172.21.15.162:0/1619964771 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f17e4199fa0 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.545+0000 7f17e9483700 1 -- 172.21.15.162:0/1619964771 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f17e419daf0 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.546+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17d4004c80 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.546+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f17d40018b0 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.546+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f17d4005280 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.546+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7f17d4006ad0 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.546+0000 7f17e9483700 1 -- 172.21.15.162:0/1619964771 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17d0005150 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.549+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f17d4005750 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.620+0000 7f17e9483700 1 -- 172.21.15.162:0/1619964771 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "status"} v 0) -- 0x7f17d0005c50 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.620+0000 7f17cbfff700 1 -- 172.21.15.162:0/1619964771 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (unknown 1155462804 0 3200959168) 0x7f17d40059b0 con 0x7f17e4104fa0 2024-04-27T01:04:20.179 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.621+0000 7f17e9483700 1 -- 172.21.15.162:0/1619964771 >> v1:172.21.15.162:6789/0 conn(0x7f17e4104fa0 legacy=0x7f17e41993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:20.180 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.621+0000 7f17e9483700 1 -- 172.21.15.162:0/1619964771 shutdown_connections 2024-04-27T01:04:20.180 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.621+0000 7f17e9483700 1 -- 172.21.15.162:0/1619964771 >> 172.21.15.162:0/1619964771 conn(0x7f17e4100520 msgr2=0x7f17e4078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:20.180 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.621+0000 7f17e9483700 1 -- 172.21.15.162:0/1619964771 shutdown_connections 2024-04-27T01:04:20.180 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:19.621+0000 7f17e9483700 1 -- 172.21.15.162:0/1619964771 wait complete. 2024-04-27T01:04:20.180 INFO:teuthology.orchestra.run.smithi162.stdout:mon is available 2024-04-27T01:04:20.180 INFO:teuthology.orchestra.run.smithi162.stdout:Assimilating anything we can from ceph.conf... 2024-04-27T01:04:21.156 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout [global] 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout fsid = ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.162:6789] 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-27T01:04:21.157 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout [osd] 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.609+0000 7ff3703f8700 1 Processor -- start 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.609+0000 7ff3703f8700 1 -- start start 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.609+0000 7ff3703f8700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff36807a380 con 0x7ff36807b780 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.609+0000 7ff36e194700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7ff36807b780 0x7ff368079be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:44496/0 (socket says 172.21.15.162:44496) 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.609+0000 7ff36e194700 1 -- 172.21.15.162:0/1149785817 learned_addr learned my addr 172.21.15.162:0/1149785817 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.610+0000 7ff36d192700 1 -- 172.21.15.162:0/1149785817 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3900615058 0 0) 0x7ff36807a380 con 0x7ff36807b780 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.610+0000 7ff36d192700 1 -- 172.21.15.162:0/1149785817 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3540035b0 con 0x7ff36807b780 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.610+0000 7ff36d192700 1 -- 172.21.15.162:0/1149785817 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2259812065 0 0) 0x7ff3540035b0 con 0x7ff36807b780 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.610+0000 7ff36d192700 1 -- 172.21.15.162:0/1149785817 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36807a550 con 0x7ff36807b780 2024-04-27T01:04:21.158 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.610+0000 7ff36d192700 1 -- 172.21.15.162:0/1149785817 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7ff364001580 con 0x7ff36807b780 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.610+0000 7ff36d192700 1 -- 172.21.15.162:0/1149785817 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff364001920 con 0x7ff36807b780 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.610+0000 7ff36d192700 1 -- 172.21.15.162:0/1149785817 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7ff364001ca0 con 0x7ff36807b780 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.610+0000 7ff3703f8700 1 -- 172.21.15.162:0/1149785817 >> v1:172.21.15.162:6789/0 conn(0x7ff36807b780 legacy=0x7ff368079be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.611+0000 7ff3703f8700 1 -- 172.21.15.162:0/1149785817 shutdown_connections 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.611+0000 7ff3703f8700 1 -- 172.21.15.162:0/1149785817 >> 172.21.15.162:0/1149785817 conn(0x7ff368101670 msgr2=0x7ff368103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.611+0000 7ff3703f8700 1 -- 172.21.15.162:0/1149785817 shutdown_connections 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.611+0000 7ff3703f8700 1 -- 172.21.15.162:0/1149785817 wait complete. 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.611+0000 7ff3703f8700 1 Processor -- start 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.611+0000 7ff3703f8700 1 -- start start 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.611+0000 7ff3703f8700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff36819bcc0 con 0x7ff36807b780 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff36e194700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7ff36807b780 0x7ff36819b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:44498/0 (socket says 172.21.15.162:44498) 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff36e194700 1 -- 172.21.15.162:0/849405261 learned_addr learned my addr 172.21.15.162:0/849405261 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1399857476 0 0) 0x7ff36819bcc0 con 0x7ff36807b780 2024-04-27T01:04:21.159 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3540031f0 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3098344518 0 0) 0x7ff3540031f0 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff36819bcc0 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7ff364001480 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 994352115 0 0) 0x7ff36819bcc0 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36819be90 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff3703f8700 1 -- 172.21.15.162:0/849405261 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff36819c180 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff3703f8700 1 -- 172.21.15.162:0/849405261 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff36819fcd0 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff364001920 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7ff364004de0 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff364005350 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.612+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7ff364006b20 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.613+0000 7ff3703f8700 1 -- 172.21.15.162:0/849405261 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff34c005150 con 0x7ff36807b780 2024-04-27T01:04:21.160 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.615+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff364005820 con 0x7ff36807b780 2024-04-27T01:04:21.161 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.684+0000 7ff3703f8700 1 -- 172.21.15.162:0/849405261 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff34c002430 con 0x7ff36807b780 2024-04-27T01:04:21.161 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.687+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+318 (unknown 1213389831 0 44894925) 0x7ff364005b90 con 0x7ff36807b780 2024-04-27T01:04:21.161 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.687+0000 7ff35f7fe700 1 -- 172.21.15.162:0/849405261 <== mon.0 v1:172.21.15.162:6789/0 11 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7ff364007000 con 0x7ff36807b780 2024-04-27T01:04:21.161 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.687+0000 7ff3703f8700 1 -- 172.21.15.162:0/849405261 >> v1:172.21.15.162:6789/0 conn(0x7ff36807b780 legacy=0x7ff36819b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:21.161 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.688+0000 7ff3703f8700 1 -- 172.21.15.162:0/849405261 shutdown_connections 2024-04-27T01:04:21.161 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.688+0000 7ff3703f8700 1 -- 172.21.15.162:0/849405261 >> 172.21.15.162:0/849405261 conn(0x7ff368101670 msgr2=0x7ff36810a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:21.161 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.688+0000 7ff3703f8700 1 -- 172.21.15.162:0/849405261 shutdown_connections 2024-04-27T01:04:21.161 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:20.688+0000 7ff3703f8700 1 -- 172.21.15.162:0/849405261 wait complete. 2024-04-27T01:04:21.161 INFO:teuthology.orchestra.run.smithi162.stdout:Generating new minimal ceph.conf... 2024-04-27T01:04:22.196 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.580+0000 7fbd077a5700 1 Processor -- start 2024-04-27T01:04:22.196 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.580+0000 7fbd077a5700 1 -- start start 2024-04-27T01:04:22.196 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.580+0000 7fbd077a5700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbd00079750 con 0x7fbd00106b40 2024-04-27T01:04:22.196 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.581+0000 7fbd05541700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fbd00106b40 0x7fbd00108f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:44500/0 (socket says 172.21.15.162:44500) 2024-04-27T01:04:22.196 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.581+0000 7fbd05541700 1 -- 172.21.15.162:0/4199577046 learned_addr learned my addr 172.21.15.162:0/4199577046 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:22.196 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.581+0000 7fbcf7fff700 1 -- 172.21.15.162:0/4199577046 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119323144 0 0) 0x7fbd00079750 con 0x7fbd00106b40 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.581+0000 7fbcf7fff700 1 -- 172.21.15.162:0/4199577046 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcec0035b0 con 0x7fbd00106b40 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.581+0000 7fbcf7fff700 1 -- 172.21.15.162:0/4199577046 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1895645116 0 0) 0x7fbcec0035b0 con 0x7fbd00106b40 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.581+0000 7fbcf7fff700 1 -- 172.21.15.162:0/4199577046 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0010b660 con 0x7fbd00106b40 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.581+0000 7fbcf7fff700 1 -- 172.21.15.162:0/4199577046 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fbcfc001580 con 0x7fbd00106b40 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.581+0000 7fbcf7fff700 1 -- 172.21.15.162:0/4199577046 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7fbcfc001d20 con 0x7fbd00106b40 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.581+0000 7fbcf7fff700 1 -- 172.21.15.162:0/4199577046 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fbcfc005960 con 0x7fbd00106b40 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.582+0000 7fbd077a5700 1 -- 172.21.15.162:0/4199577046 >> v1:172.21.15.162:6789/0 conn(0x7fbd00106b40 legacy=0x7fbd00108f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.582+0000 7fbd077a5700 1 -- 172.21.15.162:0/4199577046 shutdown_connections 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.582+0000 7fbd077a5700 1 -- 172.21.15.162:0/4199577046 >> 172.21.15.162:0/4199577046 conn(0x7fbd00100530 msgr2=0x7fbd001029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.582+0000 7fbd077a5700 1 -- 172.21.15.162:0/4199577046 shutdown_connections 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.582+0000 7fbd077a5700 1 -- 172.21.15.162:0/4199577046 wait complete. 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbd077a5700 1 Processor -- start 2024-04-27T01:04:22.197 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbd077a5700 1 -- start start 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbd077a5700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbd0019a3a0 con 0x7fbd00106b40 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbd05541700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fbd00106b40 0x7fbd00199c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:44512/0 (socket says 172.21.15.162:44512) 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbd05541700 1 -- 172.21.15.162:0/1414933846 learned_addr learned my addr 172.21.15.162:0/1414933846 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734514547 0 0) 0x7fbd0019a3a0 con 0x7fbd00106b40 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcec005530 con 0x7fbd00106b40 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1967039330 0 0) 0x7fbcec005530 con 0x7fbd00106b40 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd0019a3a0 con 0x7fbd00106b40 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fbcfc001080 con 0x7fbd00106b40 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 212340757 0 0) 0x7fbd0019a3a0 con 0x7fbd00106b40 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.583+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0019a570 con 0x7fbd00106b40 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.584+0000 7fbd077a5700 1 -- 172.21.15.162:0/1414933846 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbd0019a860 con 0x7fbd00106b40 2024-04-27T01:04:22.198 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.584+0000 7fbd077a5700 1 -- 172.21.15.162:0/1414933846 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbd0019e3b0 con 0x7fbd00106b40 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.584+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7fbcfc004ab0 con 0x7fbd00106b40 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.584+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fbcfc005f30 con 0x7fbd00106b40 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.584+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbcfc006520 con 0x7fbd00106b40 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.584+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7fbcfc007c20 con 0x7fbd00106b40 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.585+0000 7fbd077a5700 1 -- 172.21.15.162:0/1414933846 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce4005120 con 0x7fbd00106b40 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.587+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbcfc001bc0 con 0x7fbd00106b40 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.654+0000 7fbd077a5700 1 -- 172.21.15.162:0/1414933846 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fbce4005410 con 0x7fbd00106b40 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.654+0000 7fbcf67fc700 1 -- 172.21.15.162:0/1414933846 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+148 (unknown 1452402520 0 1996222575) 0x7fbcfc0056d0 con 0x7fbd00106b40 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.655+0000 7fbd077a5700 1 -- 172.21.15.162:0/1414933846 >> v1:172.21.15.162:6789/0 conn(0x7fbd00106b40 legacy=0x7fbd00199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.655+0000 7fbd077a5700 1 -- 172.21.15.162:0/1414933846 shutdown_connections 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.655+0000 7fbd077a5700 1 -- 172.21.15.162:0/1414933846 >> 172.21.15.162:0/1414933846 conn(0x7fbd00100530 msgr2=0x7fbd001908a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.655+0000 7fbd077a5700 1 -- 172.21.15.162:0/1414933846 shutdown_connections 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:21.655+0000 7fbd077a5700 1 -- 172.21.15.162:0/1414933846 wait complete. 2024-04-27T01:04:22.199 INFO:teuthology.orchestra.run.smithi162.stdout:Restarting the monitor... 2024-04-27T01:04:22.343 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:22 smithi162 systemd[1]: Stopping Ceph mon.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:04:22.759 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:22 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a[26334]: 2024-04-27T01:04:22.439+0000 7f3da67e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:04:22.759 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:22 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a[26334]: 2024-04-27T01:04:22.439+0000 7f3da67e4700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-27T01:04:22.759 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:22 smithi162 podman[26587]: 2024-04-27 01:04:22.501985709 +0000 UTC m=+0.140545497 container died 55bd07343c4990456fbc3d2d6b4c4f3d5245fb331b2ddc353558c589fc0511d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-27T01:04:23.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:22 smithi162 podman[26587]: 2024-04-27 01:04:22.864705623 +0000 UTC m=+0.503265410 container cleanup 55bd07343c4990456fbc3d2d6b4c4f3d5245fb331b2ddc353558c589fc0511d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:04:23.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:22 smithi162 bash[26587]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a 2024-04-27T01:04:23.485 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 podman[26601]: 2024-04-27 01:04:23.124061752 +0000 UTC m=+0.627275800 container remove 55bd07343c4990456fbc3d2d6b4c4f3d5245fb331b2ddc353558c589fc0511d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:04:23.485 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-27T01:04:23.485 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 systemd[1]: Stopped Ceph mon.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:04:23.486 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 systemd[1]: Starting Ceph mon.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:04:23.754 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 podman[26695]: 2024-04-27 01:04:23.491886154 +0000 UTC m=+0.103818417 container create cc5d5433d217df96e4e2fd83d028ebf1a04bcd02ee47a0f49f909f20a02fdbb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-04-27T01:04:23.754 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 podman[26695]: 2024-04-27 01:04:23.406029758 +0000 UTC m=+0.017961971 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 podman[26695]: 2024-04-27 01:04:23.669116602 +0000 UTC m=+0.281048806 container init cc5d5433d217df96e4e2fd83d028ebf1a04bcd02ee47a0f49f909f20a02fdbb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5) 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 podman[26695]: 2024-04-27 01:04:23.671935022 +0000 UTC m=+0.283867224 container start cc5d5433d217df96e4e2fd83d028ebf1a04bcd02ee47a0f49f909f20a02fdbb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev), process ceph-mon, pid 2 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: pidfile_write: ignore empty --pid-file 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: load: jerasure load: lrc 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: RocksDB version: 7.9.2 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Git sha 0 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Compile date 2024-04-25 16:43:49 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: DB SUMMARY 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: DB Session ID: TRR7D96UQZNDXAYH58HV 2024-04-27T01:04:23.755 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: CURRENT file: CURRENT 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 88863 ; 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.error_if_exists: 0 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.create_if_missing: 0 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.paranoid_checks: 1 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.env: 0x5593be41cc60 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.fs: PosixFileSystem 2024-04-27T01:04:23.756 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.info_log: 0x5593c0c1bae0 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.statistics: (nil) 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.use_fsync: 0 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_log_file_size: 0 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.allow_fallocate: 1 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.use_direct_reads: 0 2024-04-27T01:04:23.757 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.db_log_dir: 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.wal_dir: 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.write_buffer_manager: 0x5593c0bee1e0 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T01:04:23.758 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T01:04:23.759 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T01:04:23.759 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T01:04:23.759 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T01:04:23.759 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T01:04:23.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T01:04:23.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T01:04:23.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.unordered_write: 0 2024-04-27T01:04:23.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T01:04:23.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T01:04:23.760 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.row_cache: None 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.wal_filter: None 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.two_write_queues: 0 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.wal_compression: 0 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.atomic_flush: 0 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.log_readahead_size: 0 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T01:04:23.761 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_background_jobs: 2 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_background_compactions: -1 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_subcompactions: 1 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T01:04:23.762 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_open_files: -1 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_background_flushes: -1 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Compression algorithms supported: 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: kZSTD supported: 0 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: kXpressCompression supported: 0 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T01:04:23.763 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: kZlibCompression supported: 1 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: kSnappyCompression supported: 1 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: kLZ4Compression supported: 1 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: kBZip2Compression supported: 0 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.merge_operator: 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_filter: None 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T01:04:23.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5593c0c1bbe0) 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: cache_index_and_filter_blocks: 1 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: pin_top_level_index_and_filter: 1 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: index_type: 0 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: data_block_index_type: 0 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: index_shortening: 1 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: checksum: 4 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: no_block_cache: 0 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: block_cache: 0x5593c0c1f090 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: block_cache_name: BinnedLRUCache 2024-04-27T01:04:23.765 INFO:journalctl@ceph.mon.a.smithi162.stdout: block_cache_options: 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: capacity : 536870912 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: num_shard_bits : 4 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: strict_capacity_limit : 0 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: high_pri_pool_ratio: 0.000 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: block_cache_compressed: (nil) 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: persistent_cache: (nil) 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: block_size: 4096 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: block_size_deviation: 10 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: block_restart_interval: 16 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: index_block_restart_interval: 1 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: metadata_block_size: 4096 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: partition_filters: 0 2024-04-27T01:04:23.766 INFO:journalctl@ceph.mon.a.smithi162.stdout: use_delta_encoding: 1 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: filter_policy: bloomfilter 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: whole_key_filtering: 1 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: verify_compression: 0 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: read_amp_bytes_per_bit: 0 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: format_version: 5 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: enable_index_compression: 1 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: block_align: 0 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: max_auto_readahead_size: 262144 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: prepopulate_block_cache: 0 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: initial_auto_readahead_size: 8192 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout: num_file_reads_for_auto_readahead: 2 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compression: NoCompression 2024-04-27T01:04:23.767 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.num_levels: 7 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:04:23.768 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T01:04:23.769 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.arena_block_size: 1048576 2024-04-27T01:04:23.770 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.table_properties_collectors: 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.inplace_update_support: 0 2024-04-27T01:04:23.771 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.bloom_locality: 0 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.max_successive_merges: 0 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.ttl: 2592000 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-27T01:04:23.772 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.enable_blob_files: false 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.min_blob_size: 0 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.blob_file_starting_level: 0 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 47762b1f-badf-4f85-beb3-d3e9a75db0cf 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179863700925, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-27T01:04:23.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179863701777, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 85146, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 270, "table_properties": {"data_size": 83325, "index_size": 231, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10877, "raw_average_key_size": 47, "raw_value_size": 77224, "raw_average_value_size": 337, "num_data_blocks": 10, "num_entries": 229, "num_filter_entries": 229, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714179863, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "47762b1f-badf-4f85-beb3-d3e9a75db0cf", "db_session_id": "TRR7D96UQZNDXAYH58HV", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179863701857, "job": 1, "event": "recovery_finished"} 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5593c0d10000 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: DB pointer 0x5593c0cfa000 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout: ** DB Stats ** 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T01:04:23.774 INFO:journalctl@ceph.mon.a.smithi162.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: ** Compaction Stats [default] ** 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: L0 2/0 84.95 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: Sum 2/0 84.95 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: ** Compaction Stats [default] ** 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 147.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:04:23.775 INFO:journalctl@ceph.mon.a.smithi162.stdout: 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: Cumulative compaction: 0.00 GB write, 28.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: Interval compaction: 0.00 GB write, 28.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: Block cache BinnedLRUCache@0x5593c0c1f090#2 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T01:04:23.776 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: starting mon.a rank 0 at public addrs v1:172.21.15.162:6789/0 at bind addrs v1:172.21.15.162:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: mon.a@-1(???) e1 preinit fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: mon.a@-1(???).mds e1 new map 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: mon.a@-1(???).mds e1 print_map 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout: e1 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout: legacy client fscid: -1 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout: 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout: No filesystems configured 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 bash[26695]: cc5d5433d217df96e4e2fd83d028ebf1a04bcd02ee47a0f49f909f20a02fdbb7 2024-04-27T01:04:23.777 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 systemd[1]: Started Ceph mon.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: monmap epoch 1 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: last_changed 2024-04-27T01:04:16.109272+0000 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: created 2024-04-27T01:04:16.109272+0000 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: min_mon_release 19 (squid) 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: election_strategy: 1 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: 0: v1:172.21.15.162:6789/0 mon.a 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: fsmap 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: osdmap e1: 0 total, 0 up, 0 in 2024-04-27T01:04:23.778 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:23 smithi162 ceph-mon[26714]: mgrmap e1: no daemons active 2024-04-27T01:04:23.778 INFO:teuthology.orchestra.run.smithi162.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-27T01:04:24.826 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.179+0000 7f6fcb07f700 1 Processor -- start 2024-04-27T01:04:24.826 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.179+0000 7f6fcb07f700 1 -- start start 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.179+0000 7f6fcb07f700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6fc410a520 con 0x7f6fc41060f0 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.180+0000 7f6fc8e1b700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6fc41060f0 0x7f6fc4106510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:44524/0 (socket says 172.21.15.162:44524) 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.180+0000 7f6fc8e1b700 1 -- 172.21.15.162:0/2993680441 learned_addr learned my addr 172.21.15.162:0/2993680441 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.180+0000 7f6fc37fe700 1 -- 172.21.15.162:0/2993680441 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4066237231 0 0) 0x7f6fc410a520 con 0x7f6fc41060f0 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.180+0000 7f6fc37fe700 1 -- 172.21.15.162:0/2993680441 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fac0035b0 con 0x7f6fc41060f0 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.180+0000 7f6fc37fe700 1 -- 172.21.15.162:0/2993680441 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3955448826 0 0) 0x7f6fac0035b0 con 0x7f6fc41060f0 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.180+0000 7f6fc37fe700 1 -- 172.21.15.162:0/2993680441 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc410b700 con 0x7f6fc41060f0 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.180+0000 7f6fc37fe700 1 -- 172.21.15.162:0/2993680441 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f6fb4001900 con 0x7f6fc41060f0 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.180+0000 7f6fc37fe700 1 -- 172.21.15.162:0/2993680441 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f6fb40020a0 con 0x7f6fc41060f0 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.180+0000 7f6fc37fe700 1 -- 172.21.15.162:0/2993680441 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f6fb4005ce0 con 0x7f6fc41060f0 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.181+0000 7f6fcb07f700 1 -- 172.21.15.162:0/2993680441 >> v1:172.21.15.162:6789/0 conn(0x7f6fc41060f0 legacy=0x7f6fc4106510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.181+0000 7f6fcb07f700 1 -- 172.21.15.162:0/2993680441 shutdown_connections 2024-04-27T01:04:24.827 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.181+0000 7f6fcb07f700 1 -- 172.21.15.162:0/2993680441 >> 172.21.15.162:0/2993680441 conn(0x7f6fc4101670 msgr2=0x7f6fc4103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:24.828 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.181+0000 7f6fcb07f700 1 -- 172.21.15.162:0/2993680441 shutdown_connections 2024-04-27T01:04:24.828 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.181+0000 7f6fcb07f700 1 -- 172.21.15.162:0/2993680441 wait complete. 2024-04-27T01:04:24.828 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.181+0000 7f6fcb07f700 1 Processor -- start 2024-04-27T01:04:24.828 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fcb07f700 1 -- start start 2024-04-27T01:04:24.828 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fcb07f700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6fc419a340 con 0x7f6fc41060f0 2024-04-27T01:04:24.828 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fc8e1b700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6fc41060f0 0x7f6fc4199c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:44532/0 (socket says 172.21.15.162:44532) 2024-04-27T01:04:24.828 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fc8e1b700 1 -- 172.21.15.162:0/4261201462 learned_addr learned my addr 172.21.15.162:0/4261201462 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:24.828 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2482622848 0 0) 0x7f6fc419a340 con 0x7f6fc41060f0 2024-04-27T01:04:24.828 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fac005530 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2353063756 0 0) 0x7f6fac005530 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6fc419a340 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f6fb4004bf0 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3445243003 0 0) 0x7f6fc419a340 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc419a510 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.182+0000 7f6fcb07f700 1 -- 172.21.15.162:0/4261201462 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc419a800 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.183+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f6fb4004dc0 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.183+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f6fb40065d0 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.183+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6fb4006c40 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.183+0000 7f6fcb07f700 1 -- 172.21.15.162:0/4261201462 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6fc419e350 con 0x7f6fc41060f0 2024-04-27T01:04:24.829 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.183+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7f6fb40074f0 con 0x7f6fc41060f0 2024-04-27T01:04:24.830 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.183+0000 7f6fcb07f700 1 -- 172.21.15.162:0/4261201462 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc410b1f0 con 0x7f6fc41060f0 2024-04-27T01:04:24.830 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.186+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6fb40059e0 con 0x7f6fc41060f0 2024-04-27T01:04:24.830 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.252+0000 7f6fcb07f700 1 -- 172.21.15.162:0/4261201462 --> v1:172.21.15.162:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f6fc41930d0 con 0x7f6fc41060f0 2024-04-27T01:04:24.830 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.257+0000 7f6fc1ffb700 1 -- 172.21.15.162:0/4261201462 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (unknown 534127980 0 0) 0x7f6fc41930d0 con 0x7f6fc41060f0 2024-04-27T01:04:24.830 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.258+0000 7f6fcb07f700 1 -- 172.21.15.162:0/4261201462 >> v1:172.21.15.162:6789/0 conn(0x7f6fc41060f0 legacy=0x7f6fc4199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:24.830 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.258+0000 7f6fcb07f700 1 -- 172.21.15.162:0/4261201462 shutdown_connections 2024-04-27T01:04:24.830 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.258+0000 7f6fcb07f700 1 -- 172.21.15.162:0/4261201462 >> 172.21.15.162:0/4261201462 conn(0x7f6fc4101670 msgr2=0x7f6fc4103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:24.830 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.258+0000 7f6fcb07f700 1 -- 172.21.15.162:0/4261201462 shutdown_connections 2024-04-27T01:04:24.830 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:24.258+0000 7f6fcb07f700 1 -- 172.21.15.162:0/4261201462 wait complete. 2024-04-27T01:04:24.830 INFO:teuthology.orchestra.run.smithi162.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-27T01:04:24.852 INFO:teuthology.orchestra.run.smithi162.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:04:24.852 INFO:teuthology.orchestra.run.smithi162.stdout:Creating mgr... 2024-04-27T01:04:24.853 INFO:teuthology.orchestra.run.smithi162.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-27T01:04:24.853 INFO:teuthology.orchestra.run.smithi162.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-27T01:04:25.141 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mgr.y 2024-04-27T01:04:25.141 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mgr.y.service: Unit ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-27T01:04:25.289 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ea378f3c-0431-11ef-bc93-c7b262605968.target.wants/ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-ea378f3c-0431-11ef-bc93-c7b262605968@.service. 2024-04-27T01:04:25.497 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4261201462' entity='client.admin' 2024-04-27T01:04:25.497 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:25 smithi162 systemd[1]: Starting Ceph mgr.y for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:04:25.749 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:25 smithi162 podman[26928]: 2024-04-27 01:04:25.503280912 +0000 UTC m=+0.090790952 container create 17466af6dc5b9789b2a0234364343d07265050cd912a00354336d7df21295d95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb) 2024-04-27T01:04:25.749 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:25 smithi162 podman[26928]: 2024-04-27 01:04:25.430360746 +0000 UTC m=+0.017870800 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:04:25.749 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:25 smithi162 podman[26928]: 2024-04-27 01:04:25.680555554 +0000 UTC m=+0.268065594 container init 17466af6dc5b9789b2a0234364343d07265050cd912a00354336d7df21295d95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True) 2024-04-27T01:04:25.749 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:25 smithi162 podman[26928]: 2024-04-27 01:04:25.683098577 +0000 UTC m=+0.270608620 container start 17466af6dc5b9789b2a0234364343d07265050cd912a00354336d7df21295d95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-27T01:04:25.749 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:25 smithi162 bash[26928]: 17466af6dc5b9789b2a0234364343d07265050cd912a00354336d7df21295d95 2024-04-27T01:04:25.763 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T01:04:25.763 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout disabled 2024-04-27T01:04:25.772 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T01:04:25.772 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout inactive 2024-04-27T01:04:25.772 INFO:teuthology.orchestra.run.smithi162.stdout:firewalld.service is not enabled 2024-04-27T01:04:25.772 INFO:teuthology.orchestra.run.smithi162.stdout:Not possible to enable service . firewalld.service is not available 2024-04-27T01:04:25.782 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T01:04:25.782 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout disabled 2024-04-27T01:04:25.790 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T01:04:25.790 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout inactive 2024-04-27T01:04:25.791 INFO:teuthology.orchestra.run.smithi162.stdout:firewalld.service is not enabled 2024-04-27T01:04:25.791 INFO:teuthology.orchestra.run.smithi162.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-27T01:04:25.791 INFO:teuthology.orchestra.run.smithi162.stdout:Waiting for mgr to start... 2024-04-27T01:04:25.791 INFO:teuthology.orchestra.run.smithi162.stdout:Waiting for mgr... 2024-04-27T01:04:26.020 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:25 smithi162 systemd[1]: Started Ceph mgr.y for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:04:26.020 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:25 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:25.843+0000 7f64b4e0e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-27T01:04:26.334 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:26 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:26.034+0000 7f64b4e0e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:04:26.334 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:26 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:26.139+0000 7f64b4e0e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-27T01:04:26.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/891029334' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T01:04:26.593 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:26 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:26.466+0000 7f64b4e0e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-27T01:04:26.593 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:26 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:26.551+0000 7f64b4e0e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-27T01:04:26.841 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:26.844 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:04:26.844 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "fsid": "ea378f3c-0431-11ef-bc93-c7b262605968", 2024-04-27T01:04:26.844 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T01:04:26.844 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T01:04:26.844 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T01:04:26.844 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T01:04:26.844 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 0 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "a" 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T01:04:26.845 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:26.846 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T01:04:26.846 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T01:04:26.846 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T01:04:26.854 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T01:04:26.854 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T01:04:26.854 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T01:04:26.854 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:26.854 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T01:04:26.854 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T01:04:26.854 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T01:04:26.854 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T01:04:26.855 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T01:04:19.060174+0000", 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.257+0000 7f344d482700 1 Processor -- start 2024-04-27T01:04:26.856 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.258+0000 7f344d482700 1 -- start start 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.258+0000 7f344d482700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3448079690 con 0x7f3448106c30 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.258+0000 7f3446ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f3448106c30 0x7f3448109020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52812/0 (socket says 172.21.15.162:52812) 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.258+0000 7f3446ffd700 1 -- 172.21.15.162:0/3213866326 learned_addr learned my addr 172.21.15.162:0/3213866326 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.258+0000 7f3445ffb700 1 -- 172.21.15.162:0/3213866326 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2789850510 0 0) 0x7f3448079690 con 0x7f3448106c30 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.258+0000 7f3445ffb700 1 -- 172.21.15.162:0/3213866326 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34300035b0 con 0x7f3448106c30 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.259+0000 7f3445ffb700 1 -- 172.21.15.162:0/3213866326 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1818458383 0 0) 0x7f34300035b0 con 0x7f3448106c30 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.259+0000 7f3445ffb700 1 -- 172.21.15.162:0/3213866326 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344810b750 con 0x7f3448106c30 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.259+0000 7f3445ffb700 1 -- 172.21.15.162:0/3213866326 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f3438001580 con 0x7f3448106c30 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.259+0000 7f3445ffb700 1 -- 172.21.15.162:0/3213866326 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f3438001c50 con 0x7f3448106c30 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.259+0000 7f3445ffb700 1 -- 172.21.15.162:0/3213866326 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f3438005960 con 0x7f3448106c30 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.260+0000 7f344d482700 1 -- 172.21.15.162:0/3213866326 >> v1:172.21.15.162:6789/0 conn(0x7f3448106c30 legacy=0x7f3448109020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.260+0000 7f344d482700 1 -- 172.21.15.162:0/3213866326 shutdown_connections 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.260+0000 7f344d482700 1 -- 172.21.15.162:0/3213866326 >> 172.21.15.162:0/3213866326 conn(0x7f3448100560 msgr2=0x7f34481029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.260+0000 7f344d482700 1 -- 172.21.15.162:0/3213866326 shutdown_connections 2024-04-27T01:04:26.857 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.260+0000 7f344d482700 1 -- 172.21.15.162:0/3213866326 wait complete. 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.260+0000 7f344d482700 1 Processor -- start 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.261+0000 7f344d482700 1 -- start start 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.261+0000 7f344d482700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f344819a420 con 0x7f3448106c30 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.261+0000 7f3446ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f3448106c30 0x7f3448199d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52820/0 (socket says 172.21.15.162:52820) 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.261+0000 7f3446ffd700 1 -- 172.21.15.162:0/891029334 learned_addr learned my addr 172.21.15.162:0/891029334 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.261+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3142605375 0 0) 0x7f344819a420 con 0x7f3448106c30 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.261+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3430005580 con 0x7f3448106c30 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3665968183 0 0) 0x7f3430005580 con 0x7f3448106c30 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f344819a420 con 0x7f3448106c30 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f3438001080 con 0x7f3448106c30 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3293803056 0 0) 0x7f344819a420 con 0x7f3448106c30 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344819a5f0 con 0x7f3448106c30 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f344d482700 1 -- 172.21.15.162:0/891029334 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f344819a8e0 con 0x7f3448106c30 2024-04-27T01:04:26.858 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f344d482700 1 -- 172.21.15.162:0/891029334 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f344819e420 con 0x7f3448106c30 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f34380049d0 con 0x7f3448106c30 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f3438006280 con 0x7f3448106c30 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3438006890 con 0x7f3448106c30 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.262+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7f3438008010 con 0x7f3448106c30 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.263+0000 7f344d482700 1 -- 172.21.15.162:0/891029334 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3448063890 con 0x7f3448106c30 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.267+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f34380055f0 con 0x7f3448106c30 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.339+0000 7f344d482700 1 -- 172.21.15.162:0/891029334 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f344802dc80 con 0x7f3448106c30 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.340+0000 7f342ffff700 1 -- 172.21.15.162:0/891029334 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 1311435643) 0x7f3438005870 con 0x7f3448106c30 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.341+0000 7f344d482700 1 -- 172.21.15.162:0/891029334 >> v1:172.21.15.162:6789/0 conn(0x7f3448106c30 legacy=0x7f3448199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.341+0000 7f344d482700 1 -- 172.21.15.162:0/891029334 shutdown_connections 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.341+0000 7f344d482700 1 -- 172.21.15.162:0/891029334 >> 172.21.15.162:0/891029334 conn(0x7f3448100560 msgr2=0x7f34481908d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.341+0000 7f344d482700 1 -- 172.21.15.162:0/891029334 shutdown_connections 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:26.341+0000 7f344d482700 1 -- 172.21.15.162:0/891029334 wait complete. 2024-04-27T01:04:26.859 INFO:teuthology.orchestra.run.smithi162.stdout:mgr not available, waiting (1/15)... 2024-04-27T01:04:28.090 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:27 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:27.756+0000 7f64b4e0e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-27T01:04:28.091 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:27 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:27.924+0000 7f64b4e0e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-27T01:04:28.091 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:28 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:28.096+0000 7f64b4e0e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-27T01:04:28.670 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:28 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:28.381+0000 7f64b4e0e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-27T01:04:28.671 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:28 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:28.473+0000 7f64b4e0e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-27T01:04:28.671 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:28 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:28.676+0000 7f64b4e0e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-27T01:04:28.979 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:28 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:28.866+0000 7f64b4e0e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:04:29.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3720890453' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T01:04:29.593 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:29.506+0000 7f64b4e0e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-27T01:04:29.966 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:29.966 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:04:29.966 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "fsid": "ea378f3c-0431-11ef-bc93-c7b262605968", 2024-04-27T01:04:29.966 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T01:04:29.966 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T01:04:29.966 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T01:04:29.966 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T01:04:29.966 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:29.966 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 0 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "a" 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:29.967 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T01:04:29.968 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T01:04:29.969 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T01:04:19.060174+0000", 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.280+0000 7f52a5ad7700 1 Processor -- start 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.280+0000 7f52a5ad7700 1 -- start start 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.280+0000 7f52a5ad7700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f52a007a320 con 0x7f52a007b720 2024-04-27T01:04:29.970 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.281+0000 7f529f7fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f52a007b720 0x7f52a0079b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52834/0 (socket says 172.21.15.162:52834) 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.281+0000 7f529f7fe700 1 -- 172.21.15.162:0/3697052257 learned_addr learned my addr 172.21.15.162:0/3697052257 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.281+0000 7f529e7fc700 1 -- 172.21.15.162:0/3697052257 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3810810228 0 0) 0x7f52a007a320 con 0x7f52a007b720 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.281+0000 7f529e7fc700 1 -- 172.21.15.162:0/3697052257 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52880035b0 con 0x7f52a007b720 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.281+0000 7f529e7fc700 1 -- 172.21.15.162:0/3697052257 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 132210662 0 0) 0x7f52880035b0 con 0x7f52a007b720 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.281+0000 7f529e7fc700 1 -- 172.21.15.162:0/3697052257 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a007a4f0 con 0x7f52a007b720 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.281+0000 7f529e7fc700 1 -- 172.21.15.162:0/3697052257 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f5290001580 con 0x7f52a007b720 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.281+0000 7f529e7fc700 1 -- 172.21.15.162:0/3697052257 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f5290001d20 con 0x7f52a007b720 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.281+0000 7f529e7fc700 1 -- 172.21.15.162:0/3697052257 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f5290005960 con 0x7f52a007b720 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.282+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3697052257 >> v1:172.21.15.162:6789/0 conn(0x7f52a007b720 legacy=0x7f52a0079b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.282+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3697052257 shutdown_connections 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.282+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3697052257 >> 172.21.15.162:0/3697052257 conn(0x7f52a0101640 msgr2=0x7f52a0103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.282+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3697052257 shutdown_connections 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.282+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3697052257 wait complete. 2024-04-27T01:04:29.971 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.283+0000 7f52a5ad7700 1 Processor -- start 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.283+0000 7f52a5ad7700 1 -- start start 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.283+0000 7f52a5ad7700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f52a019c550 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.283+0000 7f529f7fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f52a007b720 0x7f52a019be40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52836/0 (socket says 172.21.15.162:52836) 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.283+0000 7f529f7fe700 1 -- 172.21.15.162:0/3720890453 learned_addr learned my addr 172.21.15.162:0/3720890453 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.283+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4102302919 0 0) 0x7f52a019c550 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.283+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5288005530 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.283+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2438246815 0 0) 0x7f5288005530 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.284+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52a019c550 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.284+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f5290001080 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.284+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1426121669 0 0) 0x7f52a019c550 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.284+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a019c720 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.284+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3720890453 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f52a019ca10 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.284+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3720890453 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f52a01a0560 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.284+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f5290001c80 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.284+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f5290005fa0 con 0x7f52a007b720 2024-04-27T01:04:29.972 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.284+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f52900065b0 con 0x7f52a007b720 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.284+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7f5290007cb0 con 0x7f52a007b720 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.285+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3720890453 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f528c005150 con 0x7f52a007b720 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.287+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5290004ab0 con 0x7f52a007b720 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.359+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3720890453 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f528c005440 con 0x7f52a007b720 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.360+0000 7f529cff9700 1 -- 172.21.15.162:0/3720890453 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 564797893) 0x7f5290004d70 con 0x7f52a007b720 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.361+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3720890453 >> v1:172.21.15.162:6789/0 conn(0x7f52a007b720 legacy=0x7f52a019be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.361+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3720890453 shutdown_connections 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.361+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3720890453 >> 172.21.15.162:0/3720890453 conn(0x7f52a0101640 msgr2=0x7f52a010a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.361+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3720890453 shutdown_connections 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:29.361+0000 7f52a5ad7700 1 -- 172.21.15.162:0/3720890453 wait complete. 2024-04-27T01:04:29.973 INFO:teuthology.orchestra.run.smithi162.stdout:mgr not available, waiting (2/15)... 2024-04-27T01:04:30.230 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:29.936+0000 7f64b4e0e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-27T01:04:30.230 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:30 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:30.035+0000 7f64b4e0e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-27T01:04:30.230 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:30 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:30.236+0000 7f64b4e0e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-27T01:04:31.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:30 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:30.759+0000 7f64b4e0e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-27T01:04:31.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:30 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:30.850+0000 7f64b4e0e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-27T01:04:31.342 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:31.093+0000 7f64b4e0e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-27T01:04:31.343 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:31.227+0000 7f64b4e0e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-27T01:04:31.343 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:31.322+0000 7f64b4e0e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-27T01:04:31.843 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:31.437+0000 7f64b4e0e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-27T01:04:31.843 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:31.529+0000 7f64b4e0e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-27T01:04:32.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2361624940' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T01:04:32.593 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:32 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:32.358+0000 7f64b4e0e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-27T01:04:32.593 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:32 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:32.583+0000 7f64b4e0e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-27T01:04:32.980 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:32.980 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:04:32.980 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "fsid": "ea378f3c-0431-11ef-bc93-c7b262605968", 2024-04-27T01:04:32.980 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T01:04:32.980 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 0 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "a" 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:32.981 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T01:04:32.982 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T01:04:32.983 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T01:04:19.060174+0000", 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.354+0000 7f47d6cf3700 1 Processor -- start 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.354+0000 7f47d6cf3700 1 -- start start 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.354+0000 7f47d6cf3700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47d0109410 con 0x7f47d0104fe0 2024-04-27T01:04:32.984 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.354+0000 7f47d4a8f700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f47d0104fe0 0x7f47d0105400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52846/0 (socket says 172.21.15.162:52846) 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.354+0000 7f47d4a8f700 1 -- 172.21.15.162:0/1109977163 learned_addr learned my addr 172.21.15.162:0/1109977163 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.354+0000 7f47cf7fe700 1 -- 172.21.15.162:0/1109977163 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3046606904 0 0) 0x7f47d0109410 con 0x7f47d0104fe0 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.354+0000 7f47cf7fe700 1 -- 172.21.15.162:0/1109977163 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47c00035b0 con 0x7f47d0104fe0 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.355+0000 7f47cf7fe700 1 -- 172.21.15.162:0/1109977163 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1079964186 0 0) 0x7f47c00035b0 con 0x7f47d0104fe0 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.355+0000 7f47cf7fe700 1 -- 172.21.15.162:0/1109977163 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d010a5f0 con 0x7f47d0104fe0 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.355+0000 7f47cf7fe700 1 -- 172.21.15.162:0/1109977163 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f47c4001580 con 0x7f47d0104fe0 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.355+0000 7f47cf7fe700 1 -- 172.21.15.162:0/1109977163 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f47c4001d20 con 0x7f47d0104fe0 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.355+0000 7f47cf7fe700 1 -- 172.21.15.162:0/1109977163 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f47c4005960 con 0x7f47d0104fe0 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.355+0000 7f47d6cf3700 1 -- 172.21.15.162:0/1109977163 >> v1:172.21.15.162:6789/0 conn(0x7f47d0104fe0 legacy=0x7f47d0105400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.355+0000 7f47d6cf3700 1 -- 172.21.15.162:0/1109977163 shutdown_connections 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.355+0000 7f47d6cf3700 1 -- 172.21.15.162:0/1109977163 >> 172.21.15.162:0/1109977163 conn(0x7f47d0100560 msgr2=0x7f47d01029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.355+0000 7f47d6cf3700 1 -- 172.21.15.162:0/1109977163 shutdown_connections 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.355+0000 7f47d6cf3700 1 -- 172.21.15.162:0/1109977163 wait complete. 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.356+0000 7f47d6cf3700 1 Processor -- start 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.356+0000 7f47d6cf3700 1 -- start start 2024-04-27T01:04:32.985 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.356+0000 7f47d6cf3700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f47d019a350 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.356+0000 7f47d4a8f700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f47d0104fe0 0x7f47d0199c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52862/0 (socket says 172.21.15.162:52862) 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.356+0000 7f47d4a8f700 1 -- 172.21.15.162:0/2361624940 learned_addr learned my addr 172.21.15.162:0/2361624940 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 432575525 0 0) 0x7f47d019a350 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47c0005530 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2690742073 0 0) 0x7f47c0005530 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47d019a350 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f47c4004870 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3606468892 0 0) 0x7f47d019a350 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d019a520 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47d6cf3700 1 -- 172.21.15.162:0/2361624940 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f47d019a810 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f47c4004a40 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f47c4006190 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f47c40067c0 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47d6cf3700 1 -- 172.21.15.162:0/2361624940 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f47d019e360 con 0x7f47d0104fe0 2024-04-27T01:04:32.986 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.357+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7f47c4007020 con 0x7f47d0104fe0 2024-04-27T01:04:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.358+0000 7f47d6cf3700 1 -- 172.21.15.162:0/2361624940 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d010a0e0 con 0x7f47d0104fe0 2024-04-27T01:04:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.360+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f47c4005660 con 0x7f47d0104fe0 2024-04-27T01:04:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.432+0000 7f47d6cf3700 1 -- 172.21.15.162:0/2361624940 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f47d019e870 con 0x7f47d0104fe0 2024-04-27T01:04:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.433+0000 7f47cdffb700 1 -- 172.21.15.162:0/2361624940 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 2942614020) 0x7f47c40058e0 con 0x7f47d0104fe0 2024-04-27T01:04:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.434+0000 7f47d6cf3700 1 -- 172.21.15.162:0/2361624940 >> v1:172.21.15.162:6789/0 conn(0x7f47d0104fe0 legacy=0x7f47d0199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.434+0000 7f47d6cf3700 1 -- 172.21.15.162:0/2361624940 shutdown_connections 2024-04-27T01:04:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.434+0000 7f47d6cf3700 1 -- 172.21.15.162:0/2361624940 >> 172.21.15.162:0/2361624940 conn(0x7f47d0100560 msgr2=0x7f47d00788e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.434+0000 7f47d6cf3700 1 -- 172.21.15.162:0/2361624940 shutdown_connections 2024-04-27T01:04:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:32.434+0000 7f47d6cf3700 1 -- 172.21.15.162:0/2361624940 wait complete. 2024-04-27T01:04:32.987 INFO:teuthology.orchestra.run.smithi162.stdout:mgr not available, waiting (3/15)... 2024-04-27T01:04:33.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: Activating manager daemon y 2024-04-27T01:04:33.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: mgrmap e2: y(active, starting, since 0.00299362s) 2024-04-27T01:04:33.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:04:33.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:04:33.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:04:33.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:04:33.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:04:33.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:04:33.844 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:04:33.844 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-27T01:04:33.844 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: Manager daemon y is now available 2024-04-27T01:04:33.844 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:04:33.844 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-27T01:04:33.844 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' 2024-04-27T01:04:33.844 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' 2024-04-27T01:04:33.844 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:33 smithi162 ceph-mon[26714]: from='mgr.14100 v1:172.21.15.162:0/2451815152' entity='mgr.y' 2024-04-27T01:04:34.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:34 smithi162 ceph-mon[26714]: mgrmap e3: y(active, since 1.00687s) 2024-04-27T01:04:35.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:35 smithi162 ceph-mon[26714]: mgrmap e4: y(active, since 2s) 2024-04-27T01:04:38.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1927731073' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-27T01:04:44.445 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "fsid": "ea378f3c-0431-11ef-bc93-c7b262605968", 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "health": { 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 0 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-27T01:04:44.446 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "a" 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-27T01:04:44.447 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-27T01:04:44.455 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-27T01:04:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "restful" 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ], 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "modified": "2024-04-27T01:04:19.060174+0000", 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout }, 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.993+0000 7fb004599700 1 Processor -- start 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.993+0000 7fb004599700 1 -- start start 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.993+0000 7fb004599700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faffc07a380 con 0x7faffc07b780 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.993+0000 7fb002335700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7faffc07b780 0x7faffc079be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47416/0 (socket says 172.21.15.162:47416) 2024-04-27T01:04:44.457 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.993+0000 7fb002335700 1 -- 172.21.15.162:0/3608972457 learned_addr learned my addr 172.21.15.162:0/3608972457 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.994+0000 7fb001333700 1 -- 172.21.15.162:0/3608972457 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1525350191 0 0) 0x7faffc07a380 con 0x7faffc07b780 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.994+0000 7fb001333700 1 -- 172.21.15.162:0/3608972457 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafe80035b0 con 0x7faffc07b780 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.994+0000 7fb001333700 1 -- 172.21.15.162:0/3608972457 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2815607486 0 0) 0x7fafe80035b0 con 0x7faffc07b780 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.994+0000 7fb001333700 1 -- 172.21.15.162:0/3608972457 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faffc07a550 con 0x7faffc07b780 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.994+0000 7fb001333700 1 -- 172.21.15.162:0/3608972457 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7faff8001580 con 0x7faffc07b780 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.994+0000 7fb001333700 1 -- 172.21.15.162:0/3608972457 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7faff8001d20 con 0x7faffc07b780 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.994+0000 7fb001333700 1 -- 172.21.15.162:0/3608972457 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7faff8005960 con 0x7faffc07b780 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.994+0000 7fb004599700 1 -- 172.21.15.162:0/3608972457 >> v1:172.21.15.162:6789/0 conn(0x7faffc07b780 legacy=0x7faffc079be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.995+0000 7fb004599700 1 -- 172.21.15.162:0/3608972457 shutdown_connections 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.995+0000 7fb004599700 1 -- 172.21.15.162:0/3608972457 >> 172.21.15.162:0/3608972457 conn(0x7faffc101640 msgr2=0x7faffc103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.995+0000 7fb004599700 1 -- 172.21.15.162:0/3608972457 shutdown_connections 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.995+0000 7fb004599700 1 -- 172.21.15.162:0/3608972457 wait complete. 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.995+0000 7fb004599700 1 Processor -- start 2024-04-27T01:04:44.458 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.995+0000 7fb004599700 1 -- start start 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.995+0000 7fb004599700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7faffc19c5b0 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.995+0000 7fb002335700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7faffc07b780 0x7faffc19bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47428/0 (socket says 172.21.15.162:47428) 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.995+0000 7fb002335700 1 -- 172.21.15.162:0/1927731073 learned_addr learned my addr 172.21.15.162:0/1927731073 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364898562 0 0) 0x7faffc19c5b0 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafe8005530 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2709274918 0 0) 0x7fafe8005530 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faffc19c5b0 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7faff8001080 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 110591105 0 0) 0x7faffc19c5b0 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faffc19c780 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7fb004599700 1 -- 172.21.15.162:0/1927731073 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faffc19ca70 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7fb004599700 1 -- 172.21.15.162:0/1927731073 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faffc1a05c0 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7faff8001ba0 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.996+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7faff8005eb0 con 0x7faffc07b780 2024-04-27T01:04:44.459 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.997+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) ==== 46241+0+0 (unknown 2128764644 0 0) 0x7faff8011620 con 0x7faffc07b780 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.997+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7faff8048620 con 0x7faffc07b780 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:37.997+0000 7fb004599700 1 -- 172.21.15.162:0/1927731073 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe0005150 con 0x7faffc07b780 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:38.002+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7faff8017520 con 0x7faffc07b780 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:38.278+0000 7fb004599700 1 -- 172.21.15.162:0/1927731073 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fafe0005440 con 0x7faffc07b780 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:38.279+0000 7faff37fe700 1 -- 172.21.15.162:0/1927731073 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (unknown 4201413639 0 3450978579) 0x7faff8016e20 con 0x7faffc07b780 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:38.281+0000 7fb004599700 1 -- 172.21.15.162:0/1927731073 >> v1:172.21.15.162:6800/629135069 conn(0x7fafe803a420 legacy=0x7fafe803c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:38.281+0000 7fb004599700 1 -- 172.21.15.162:0/1927731073 >> v1:172.21.15.162:6789/0 conn(0x7faffc07b780 legacy=0x7faffc19bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:38.281+0000 7fb004599700 1 -- 172.21.15.162:0/1927731073 shutdown_connections 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:38.281+0000 7fb004599700 1 -- 172.21.15.162:0/1927731073 >> 172.21.15.162:0/1927731073 conn(0x7faffc101640 msgr2=0x7faffc10a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:38.281+0000 7fb004599700 1 -- 172.21.15.162:0/1927731073 shutdown_connections 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:38.281+0000 7fb004599700 1 -- 172.21.15.162:0/1927731073 wait complete. 2024-04-27T01:04:44.460 INFO:teuthology.orchestra.run.smithi162.stdout:mgr is available 2024-04-27T01:04:46.146 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2243337145' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-27T01:04:46.433 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:46.439 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout [global] 2024-04-27T01:04:46.439 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout fsid = ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:04:46.439 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-27T01:04:46.439 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-27T01:04:46.439 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-27T01:04:46.439 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-27T01:04:46.439 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout [osd] 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.650+0000 7f2d89d11700 1 Processor -- start 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.650+0000 7f2d89d11700 1 -- start start 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.650+0000 7f2d89d11700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2d8410a530 con 0x7f2d84106100 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.650+0000 7f2d837fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f2d84106100 0x7f2d84106520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:37488/0 (socket says 172.21.15.162:37488) 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.650+0000 7f2d837fe700 1 -- 172.21.15.162:0/2653589311 learned_addr learned my addr 172.21.15.162:0/2653589311 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.650+0000 7f2d827fc700 1 -- 172.21.15.162:0/2653589311 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226483800 0 0) 0x7f2d8410a530 con 0x7f2d84106100 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.650+0000 7f2d827fc700 1 -- 172.21.15.162:0/2653589311 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d6c0035b0 con 0x7f2d84106100 2024-04-27T01:04:46.440 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.651+0000 7f2d827fc700 1 -- 172.21.15.162:0/2653589311 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1627484666 0 0) 0x7f2d6c0035b0 con 0x7f2d84106100 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.651+0000 7f2d827fc700 1 -- 172.21.15.162:0/2653589311 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8410b710 con 0x7f2d84106100 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.651+0000 7f2d827fc700 1 -- 172.21.15.162:0/2653589311 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f2d78001580 con 0x7f2d84106100 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.651+0000 7f2d827fc700 1 -- 172.21.15.162:0/2653589311 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f2d78001d20 con 0x7f2d84106100 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.651+0000 7f2d827fc700 1 -- 172.21.15.162:0/2653589311 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f2d78005960 con 0x7f2d84106100 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.651+0000 7f2d89d11700 1 -- 172.21.15.162:0/2653589311 >> v1:172.21.15.162:6789/0 conn(0x7f2d84106100 legacy=0x7f2d84106520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.651+0000 7f2d89d11700 1 -- 172.21.15.162:0/2653589311 shutdown_connections 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.651+0000 7f2d89d11700 1 -- 172.21.15.162:0/2653589311 >> 172.21.15.162:0/2653589311 conn(0x7f2d841016a0 msgr2=0x7f2d84103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.651+0000 7f2d89d11700 1 -- 172.21.15.162:0/2653589311 shutdown_connections 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.651+0000 7f2d89d11700 1 -- 172.21.15.162:0/2653589311 wait complete. 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.652+0000 7f2d89d11700 1 Processor -- start 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.652+0000 7f2d89d11700 1 -- start start 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.652+0000 7f2d89d11700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2d8419a300 con 0x7f2d84106100 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.652+0000 7f2d837fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f2d84106100 0x7f2d84199bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:37500/0 (socket says 172.21.15.162:37500) 2024-04-27T01:04:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.652+0000 7f2d837fe700 1 -- 172.21.15.162:0/2243337145 learned_addr learned my addr 172.21.15.162:0/2243337145 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458214407 0 0) 0x7f2d8419a300 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d6c005530 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2438143339 0 0) 0x7f2d6c005530 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2d8419a300 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f2d78001080 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3733588858 0 0) 0x7f2d8419a300 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8419a4d0 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d89d11700 1 -- 172.21.15.162:0/2243337145 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8419a7c0 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d89d11700 1 -- 172.21.15.162:0/2243337145 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2d8419e310 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f2d78001900 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.653+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f2d78006040 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.654+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) ==== 46241+0+0 (unknown 2128764644 0 0) 0x7f2d78006200 con 0x7f2d84106100 2024-04-27T01:04:46.442 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.654+0000 7f2d89d11700 1 -- 172.21.15.162:0/2243337145 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d70005150 con 0x7f2d84106100 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.654+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7f2d78047770 con 0x7f2d84106100 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.659+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f2d78005950 con 0x7f2d84106100 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.854+0000 7f2d89d11700 1 -- 172.21.15.162:0/2243337145 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2d70002430 con 0x7f2d84106100 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.854+0000 7f2d80ff9700 1 -- 172.21.15.162:0/2243337145 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+282 (unknown 1187553405 0 203373214) 0x7f2d78017550 con 0x7f2d84106100 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.857+0000 7f2d89d11700 1 -- 172.21.15.162:0/2243337145 >> v1:172.21.15.162:6800/629135069 conn(0x7f2d6c03a410 legacy=0x7f2d6c03c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.857+0000 7f2d89d11700 1 -- 172.21.15.162:0/2243337145 >> v1:172.21.15.162:6789/0 conn(0x7f2d84106100 legacy=0x7f2d84199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.857+0000 7f2d89d11700 1 -- 172.21.15.162:0/2243337145 shutdown_connections 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.857+0000 7f2d89d11700 1 -- 172.21.15.162:0/2243337145 >> 172.21.15.162:0/2243337145 conn(0x7f2d841016a0 msgr2=0x7f2d84103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.857+0000 7f2d89d11700 1 -- 172.21.15.162:0/2243337145 shutdown_connections 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:45.857+0000 7f2d89d11700 1 -- 172.21.15.162:0/2243337145 wait complete. 2024-04-27T01:04:46.443 INFO:teuthology.orchestra.run.smithi162.stdout:Enabling cephadm module... 2024-04-27T01:04:47.518 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/819100685' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-27T01:04:47.519 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:47 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ignoring --setuser ceph since I am not root 2024-04-27T01:04:47.519 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:47 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ignoring --setgroup ceph since I am not root 2024-04-27T01:04:47.519 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:47 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:47.464+0000 7f83d52f0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-27T01:04:47.784 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:47 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:47.648+0000 7f83d52f0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:04:47.784 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:47 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:47.752+0000 7f83d52f0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-27T01:04:48.070 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.982+0000 7f21f86e5700 1 Processor -- start 2024-04-27T01:04:48.070 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.982+0000 7f21f86e5700 1 -- start start 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.983+0000 7f21f86e5700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21f010a530 con 0x7f21f0106100 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.983+0000 7f21f6481700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f21f0106100 0x7f21f0106520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:37516/0 (socket says 172.21.15.162:37516) 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.983+0000 7f21f6481700 1 -- 172.21.15.162:0/84326988 learned_addr learned my addr 172.21.15.162:0/84326988 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.983+0000 7f21f547f700 1 -- 172.21.15.162:0/84326988 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3535659019 0 0) 0x7f21f010a530 con 0x7f21f0106100 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.983+0000 7f21f547f700 1 -- 172.21.15.162:0/84326988 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21dc0035b0 con 0x7f21f0106100 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.983+0000 7f21f547f700 1 -- 172.21.15.162:0/84326988 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3276759313 0 0) 0x7f21dc0035b0 con 0x7f21f0106100 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.983+0000 7f21f547f700 1 -- 172.21.15.162:0/84326988 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f010b710 con 0x7f21f0106100 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.983+0000 7f21f547f700 1 -- 172.21.15.162:0/84326988 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f21e8001900 con 0x7f21f0106100 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.984+0000 7f21f547f700 1 -- 172.21.15.162:0/84326988 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f21e80020a0 con 0x7f21f0106100 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.984+0000 7f21f547f700 1 -- 172.21.15.162:0/84326988 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f21e8005ce0 con 0x7f21f0106100 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.984+0000 7f21f86e5700 1 -- 172.21.15.162:0/84326988 >> v1:172.21.15.162:6789/0 conn(0x7f21f0106100 legacy=0x7f21f0106520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:48.071 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.984+0000 7f21f86e5700 1 -- 172.21.15.162:0/84326988 shutdown_connections 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.984+0000 7f21f86e5700 1 -- 172.21.15.162:0/84326988 >> 172.21.15.162:0/84326988 conn(0x7f21f01016a0 msgr2=0x7f21f0103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.984+0000 7f21f86e5700 1 -- 172.21.15.162:0/84326988 shutdown_connections 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.984+0000 7f21f86e5700 1 -- 172.21.15.162:0/84326988 wait complete. 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.985+0000 7f21f86e5700 1 Processor -- start 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.985+0000 7f21f86e5700 1 -- start start 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.985+0000 7f21f86e5700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f21f019a300 con 0x7f21f0106100 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.985+0000 7f21f6481700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f21f0106100 0x7f21f0199bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:37524/0 (socket says 172.21.15.162:37524) 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.985+0000 7f21f6481700 1 -- 172.21.15.162:0/819100685 learned_addr learned my addr 172.21.15.162:0/819100685 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.985+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 494663439 0 0) 0x7f21f019a300 con 0x7f21f0106100 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.985+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21dc005530 con 0x7f21f0106100 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.985+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3473367997 0 0) 0x7f21dc005530 con 0x7f21f0106100 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.985+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21f019a300 con 0x7f21f0106100 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.985+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f21e8001400 con 0x7f21f0106100 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.986+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1549891953 0 0) 0x7f21f019a300 con 0x7f21f0106100 2024-04-27T01:04:48.072 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.986+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f019a4d0 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.986+0000 7f21f86e5700 1 -- 172.21.15.162:0/819100685 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f21f019a7c0 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.986+0000 7f21f86e5700 1 -- 172.21.15.162:0/819100685 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f21f019e310 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.986+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f21e8001c80 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.986+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f21e80063b0 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.986+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) ==== 46241+0+0 (unknown 2128764644 0 0) 0x7f21e8011a20 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.986+0000 7f21f86e5700 1 -- 172.21.15.162:0/819100685 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d4005150 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.987+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7f21e8047b40 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:46.991+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f21e8017900 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:47.242+0000 7f21f86e5700 1 -- 172.21.15.162:0/819100685 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f21d4005440 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:47.293+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (unknown 2263024820 0 0) 0x7f21e8005cd0 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:47.295+0000 7f21e77fe700 1 -- 172.21.15.162:0/819100685 <== mon.0 v1:172.21.15.162:6789/0 11 ==== mgrmap(e 5) ==== 46252+0+0 (unknown 1295096627 0 0) 0x7f21e8046530 con 0x7f21f0106100 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:47.296+0000 7f21f86e5700 1 -- 172.21.15.162:0/819100685 >> v1:172.21.15.162:6800/629135069 conn(0x7f21dc03a440 legacy=0x7f21dc03c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:48.073 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:47.296+0000 7f21f86e5700 1 -- 172.21.15.162:0/819100685 >> v1:172.21.15.162:6789/0 conn(0x7f21f0106100 legacy=0x7f21f0199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:48.074 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:47.296+0000 7f21f86e5700 1 -- 172.21.15.162:0/819100685 shutdown_connections 2024-04-27T01:04:48.074 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:47.296+0000 7f21f86e5700 1 -- 172.21.15.162:0/819100685 >> 172.21.15.162:0/819100685 conn(0x7f21f01016a0 msgr2=0x7f21f0103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:48.074 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:47.296+0000 7f21f86e5700 1 -- 172.21.15.162:0/819100685 shutdown_connections 2024-04-27T01:04:48.074 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:47.296+0000 7f21f86e5700 1 -- 172.21.15.162:0/819100685 wait complete. 2024-04-27T01:04:48.309 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/819100685' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-27T01:04:48.309 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:48 smithi162 ceph-mon[26714]: mgrmap e5: y(active, since 14s) 2024-04-27T01:04:48.310 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:48 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:48.062+0000 7f83d52f0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-27T01:04:48.310 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:48 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:48.152+0000 7f83d52f0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-27T01:04:49.303 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1815571673' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-27T01:04:49.311 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:04:49.311 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-27T01:04:49.311 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-27T01:04:49.311 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.490+0000 7f20e6d4a700 1 Processor -- start 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.490+0000 7f20e6d4a700 1 -- start start 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.490+0000 7f20e6d4a700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f20e0079710 con 0x7f20e0106b60 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.490+0000 7f20e4ae6700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f20e0106b60 0x7f20e0108f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:37552/0 (socket says 172.21.15.162:37552) 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.490+0000 7f20e4ae6700 1 -- 172.21.15.162:0/2616170252 learned_addr learned my addr 172.21.15.162:0/2616170252 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.491+0000 7f20df7fe700 1 -- 172.21.15.162:0/2616170252 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2114902330 0 0) 0x7f20e0079710 con 0x7f20e0106b60 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.491+0000 7f20df7fe700 1 -- 172.21.15.162:0/2616170252 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d00035b0 con 0x7f20e0106b60 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.491+0000 7f20df7fe700 1 -- 172.21.15.162:0/2616170252 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4210921333 0 0) 0x7f20d00035b0 con 0x7f20e0106b60 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.491+0000 7f20df7fe700 1 -- 172.21.15.162:0/2616170252 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e010b680 con 0x7f20e0106b60 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.491+0000 7f20df7fe700 1 -- 172.21.15.162:0/2616170252 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f20d4001580 con 0x7f20e0106b60 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.491+0000 7f20df7fe700 1 -- 172.21.15.162:0/2616170252 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f20d4001c50 con 0x7f20e0106b60 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.491+0000 7f20df7fe700 1 -- 172.21.15.162:0/2616170252 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f20d4005960 con 0x7f20e0106b60 2024-04-27T01:04:49.312 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.491+0000 7f20e6d4a700 1 -- 172.21.15.162:0/2616170252 >> v1:172.21.15.162:6789/0 conn(0x7f20e0106b60 legacy=0x7f20e0108f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.492+0000 7f20e6d4a700 1 -- 172.21.15.162:0/2616170252 shutdown_connections 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.492+0000 7f20e6d4a700 1 -- 172.21.15.162:0/2616170252 >> 172.21.15.162:0/2616170252 conn(0x7f20e0100530 msgr2=0x7f20e0102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.492+0000 7f20e6d4a700 1 -- 172.21.15.162:0/2616170252 shutdown_connections 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.492+0000 7f20e6d4a700 1 -- 172.21.15.162:0/2616170252 wait complete. 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.492+0000 7f20e6d4a700 1 Processor -- start 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.492+0000 7f20e6d4a700 1 -- start start 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.492+0000 7f20e6d4a700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f20e019a410 con 0x7f20e0106b60 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.492+0000 7f20e4ae6700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f20e0106b60 0x7f20e0199d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:37568/0 (socket says 172.21.15.162:37568) 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.492+0000 7f20e4ae6700 1 -- 172.21.15.162:0/1815571673 learned_addr learned my addr 172.21.15.162:0/1815571673 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296672417 0 0) 0x7f20e019a410 con 0x7f20e0106b60 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d0005580 con 0x7f20e0106b60 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2091610606 0 0) 0x7f20d0005580 con 0x7f20e0106b60 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f20e019a410 con 0x7f20e0106b60 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f20d4001080 con 0x7f20e0106b60 2024-04-27T01:04:49.313 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2689716991 0 0) 0x7f20e019a410 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e019a5e0 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20e6d4a700 1 -- 172.21.15.162:0/1815571673 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f20e019a8d0 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f20d40049d0 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f20d4006280 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.493+0000 7f20e6d4a700 1 -- 172.21.15.162:0/1815571673 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f20e019e410 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.494+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 5) ==== 46252+0+0 (unknown 1295096627 0 0) 0x7f20d4011a30 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.494+0000 7f20e6d4a700 1 -- 172.21.15.162:0/1815571673 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e0063890 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.495+0000 7f20dffff700 1 -- 172.21.15.162:0/1815571673 >> v1:172.21.15.162:6800/629135069 conn(0x7f20d003a460 legacy=0x7f20d003c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/629135069 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.495+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7f20d4047840 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.499+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f20d40175c0 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.696+0000 7f20dffff700 1 -- 172.21.15.162:0/1815571673 >> v1:172.21.15.162:6800/629135069 conn(0x7f20d003a460 legacy=0x7f20d003c920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/629135069 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.791+0000 7f20e6d4a700 1 -- 172.21.15.162:0/1815571673 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f20e002dc30 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.792+0000 7f20ddffb700 1 -- 172.21.15.162:0/1815571673 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (unknown 3768197548 0 15966916) 0x7f20d4016ec0 con 0x7f20e0106b60 2024-04-27T01:04:49.314 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.795+0000 7f20e6d4a700 1 -- 172.21.15.162:0/1815571673 >> v1:172.21.15.162:6800/629135069 conn(0x7f20d003a460 legacy=0x7f20d003c920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:04:49.315 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.795+0000 7f20e6d4a700 1 -- 172.21.15.162:0/1815571673 >> v1:172.21.15.162:6789/0 conn(0x7f20e0106b60 legacy=0x7f20e0199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:49.315 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.795+0000 7f20e6d4a700 1 -- 172.21.15.162:0/1815571673 shutdown_connections 2024-04-27T01:04:49.315 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.795+0000 7f20e6d4a700 1 -- 172.21.15.162:0/1815571673 >> 172.21.15.162:0/1815571673 conn(0x7f20e0100530 msgr2=0x7f20e0190880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:49.315 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.795+0000 7f20e6d4a700 1 -- 172.21.15.162:0/1815571673 shutdown_connections 2024-04-27T01:04:49.315 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:48.795+0000 7f20e6d4a700 1 -- 172.21.15.162:0/1815571673 wait complete. 2024-04-27T01:04:49.315 INFO:teuthology.orchestra.run.smithi162.stdout:Waiting for the mgr to restart... 2024-04-27T01:04:49.315 INFO:teuthology.orchestra.run.smithi162.stdout:Waiting for mgr epoch 5... 2024-04-27T01:04:49.592 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:49.378+0000 7f83d52f0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-27T01:04:49.593 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:49.551+0000 7f83d52f0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-27T01:04:49.927 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:49.727+0000 7f83d52f0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-27T01:04:50.307 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:50 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:50.017+0000 7f83d52f0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-27T01:04:50.307 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:50 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:50.109+0000 7f83d52f0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-27T01:04:50.307 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:50 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:50.313+0000 7f83d52f0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-27T01:04:50.592 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:50 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:50.508+0000 7f83d52f0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:04:51.369 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:51.112+0000 7f83d52f0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-27T01:04:51.632 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:51.543+0000 7f83d52f0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-27T01:04:51.632 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:51.639+0000 7f83d52f0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-27T01:04:52.092 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:51.840+0000 7f83d52f0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-27T01:04:52.687 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:52.362+0000 7f83d52f0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-27T01:04:52.687 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:52.455+0000 7f83d52f0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-27T01:04:53.027 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:52.694+0000 7f83d52f0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-27T01:04:53.028 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:52.827+0000 7f83d52f0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-27T01:04:53.028 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:52.921+0000 7f83d52f0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-27T01:04:53.028 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:53 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:53.034+0000 7f83d52f0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-27T01:04:53.342 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:53 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:53.126+0000 7f83d52f0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-27T01:04:54.199 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:53 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:53.939+0000 7f83d52f0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-27T01:04:54.199 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:04:54.153+0000 7f83d52f0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-27T01:04:54.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: Active manager daemon y restarted 2024-04-27T01:04:54.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: Activating manager daemon y 2024-04-27T01:04:54.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: osdmap e2: 0 total, 0 up, 0 in 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: mgrmap e6: y(active, starting, since 0.00310897s) 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: Manager daemon y is now available 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:04:54.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:54 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-27T01:04:55.442 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:55 smithi162 ceph-mon[26714]: Found migration_current of "None". Setting to last migration. 2024-04-27T01:04:55.442 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:55 smithi162 ceph-mon[26714]: mgrmap e7: y(active, since 1.00508s) 2024-04-27T01:04:55.717 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:04:55.717 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-27T01:04:55.717 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-27T01:04:55.717 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:04:55.717 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.729+0000 7ffa263dc700 1 Processor -- start 2024-04-27T01:04:55.717 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.729+0000 7ffa263dc700 1 -- start start 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.729+0000 7ffa263dc700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffa2010a550 con 0x7ffa20104390 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.729+0000 7ffa1ffff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7ffa20104390 0x7ffa201047b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:37572/0 (socket says 172.21.15.162:37572) 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.729+0000 7ffa1ffff700 1 -- 172.21.15.162:0/604411016 learned_addr learned my addr 172.21.15.162:0/604411016 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.729+0000 7ffa1effd700 1 -- 172.21.15.162:0/604411016 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2396158926 0 0) 0x7ffa2010a550 con 0x7ffa20104390 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.729+0000 7ffa1effd700 1 -- 172.21.15.162:0/604411016 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa080035b0 con 0x7ffa20104390 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.730+0000 7ffa1effd700 1 -- 172.21.15.162:0/604411016 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1384362778 0 0) 0x7ffa080035b0 con 0x7ffa20104390 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.730+0000 7ffa1effd700 1 -- 172.21.15.162:0/604411016 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2010b730 con 0x7ffa20104390 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.730+0000 7ffa1effd700 1 -- 172.21.15.162:0/604411016 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7ffa10001580 con 0x7ffa20104390 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.730+0000 7ffa1effd700 1 -- 172.21.15.162:0/604411016 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7ffa10001d20 con 0x7ffa20104390 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.730+0000 7ffa1effd700 1 -- 172.21.15.162:0/604411016 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7ffa10005960 con 0x7ffa20104390 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.730+0000 7ffa263dc700 1 -- 172.21.15.162:0/604411016 >> v1:172.21.15.162:6789/0 conn(0x7ffa20104390 legacy=0x7ffa201047b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.730+0000 7ffa263dc700 1 -- 172.21.15.162:0/604411016 shutdown_connections 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.730+0000 7ffa263dc700 1 -- 172.21.15.162:0/604411016 >> 172.21.15.162:0/604411016 conn(0x7ffa200ffca0 msgr2=0x7ffa20102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:55.718 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.730+0000 7ffa263dc700 1 -- 172.21.15.162:0/604411016 shutdown_connections 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.731+0000 7ffa263dc700 1 -- 172.21.15.162:0/604411016 wait complete. 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.731+0000 7ffa263dc700 1 Processor -- start 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.731+0000 7ffa263dc700 1 -- start start 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.731+0000 7ffa263dc700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ffa2019fce0 con 0x7ffa20104390 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.731+0000 7ffa1ffff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7ffa20104390 0x7ffa20106ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:37586/0 (socket says 172.21.15.162:37586) 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.731+0000 7ffa1ffff700 1 -- 172.21.15.162:0/4089738458 learned_addr learned my addr 172.21.15.162:0/4089738458 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4278445057 0 0) 0x7ffa2019fce0 con 0x7ffa20104390 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa08005530 con 0x7ffa20104390 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1859154258 0 0) 0x7ffa08005530 con 0x7ffa20104390 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffa2019fce0 con 0x7ffa20104390 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7ffa10001080 con 0x7ffa20104390 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1675712852 0 0) 0x7ffa2019fce0 con 0x7ffa20104390 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa201a0ec0 con 0x7ffa20104390 2024-04-27T01:04:55.719 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa263dc700 1 -- 172.21.15.162:0/4089738458 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffa2019ff10 con 0x7ffa20104390 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa263dc700 1 -- 172.21.15.162:0/4089738458 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ffa201a0450 con 0x7ffa20104390 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7ffa10004ab0 con 0x7ffa20104390 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.732+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7ffa10005ec0 con 0x7ffa20104390 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.733+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 5) ==== 46252+0+0 (unknown 1295096627 0 0) 0x7ffa10011520 con 0x7ffa20104390 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.733+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 --> v1:172.21.15.162:6800/629135069 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffa10001080 con 0x7ffa0803a4c0 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.733+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 77045045 0 0) 0x7ffa10047620 con 0x7ffa20104390 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.733+0000 7ffa1f7fe700 1 -- 172.21.15.162:0/4089738458 >> v1:172.21.15.162:6800/629135069 conn(0x7ffa0803a4c0 legacy=0x7ffa0803c980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/629135069 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:49.934+0000 7ffa1f7fe700 1 -- 172.21.15.162:0/4089738458 >> v1:172.21.15.162:6800/629135069 conn(0x7ffa0803a4c0 legacy=0x7ffa0803c980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/629135069 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:50.335+0000 7ffa1f7fe700 1 -- 172.21.15.162:0/4089738458 >> v1:172.21.15.162:6800/629135069 conn(0x7ffa0803a4c0 legacy=0x7ffa0803c980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/629135069 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:51.136+0000 7ffa1f7fe700 1 -- 172.21.15.162:0/4089738458 >> v1:172.21.15.162:6800/629135069 conn(0x7ffa0803a4c0 legacy=0x7ffa0803c980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/629135069 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:52.737+0000 7ffa1f7fe700 1 -- 172.21.15.162:0/4089738458 >> v1:172.21.15.162:6800/629135069 conn(0x7ffa0803a4c0 legacy=0x7ffa0803c980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/629135069 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:54.157+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mgrmap(e 6) ==== 46054+0+0 (unknown 2691767739 0 0) 0x7ffa10013370 con 0x7ffa20104390 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:54.157+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 >> v1:172.21.15.162:6800/629135069 conn(0x7ffa0803a4c0 legacy=0x7ffa0803c980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.158+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mgrmap(e 7) ==== 46146+0+0 (unknown 2026620879 0 0) 0x7ffa10048940 con 0x7ffa20104390 2024-04-27T01:04:55.720 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.158+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 --> v1:172.21.15.162:6800/106205076 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ffa10001080 con 0x7ffa0803ed50 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.159+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mgr.14120 v1:172.21.15.162:6800/106205076 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (unknown 0 0 4109177037) 0x7ffa10001080 con 0x7ffa0803ed50 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.165+0000 7ffa263dc700 1 -- 172.21.15.162:0/4089738458 --> v1:172.21.15.162:6800/106205076 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ffa0c002630 con 0x7ffa0803ed50 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.165+0000 7ffa1d7fa700 1 -- 172.21.15.162:0/4089738458 <== mgr.14120 v1:172.21.15.162:6800/106205076 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (unknown 0 0 96372106) 0x7ffa0c002630 con 0x7ffa0803ed50 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.166+0000 7ffa263dc700 1 -- 172.21.15.162:0/4089738458 >> v1:172.21.15.162:6800/106205076 conn(0x7ffa0803ed50 legacy=0x7ffa08041140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.166+0000 7ffa263dc700 1 -- 172.21.15.162:0/4089738458 >> v1:172.21.15.162:6789/0 conn(0x7ffa20104390 legacy=0x7ffa20106ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.166+0000 7ffa263dc700 1 -- 172.21.15.162:0/4089738458 shutdown_connections 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.166+0000 7ffa263dc700 1 -- 172.21.15.162:0/4089738458 >> 172.21.15.162:0/4089738458 conn(0x7ffa200ffca0 msgr2=0x7ffa201908b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.166+0000 7ffa263dc700 1 -- 172.21.15.162:0/4089738458 shutdown_connections 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:55.166+0000 7ffa263dc700 1 -- 172.21.15.162:0/4089738458 wait complete. 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:mgr epoch 5 is available 2024-04-27T01:04:55.721 INFO:teuthology.orchestra.run.smithi162.stdout:Setting orchestrator backend to cephadm... 2024-04-27T01:04:56.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:56 smithi162 ceph-mon[26714]: from='client.14124 v1:172.21.15.162:0/4089738458' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-27T01:04:56.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:56 smithi162 ceph-mon[26714]: from='client.14124 v1:172.21.15.162:0/4089738458' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-27T01:04:56.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:56 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:04:56.343 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:56 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:04:57.016 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.178+0000 7f890b3f8700 1 Processor -- start 2024-04-27T01:04:57.016 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.179+0000 7f890b3f8700 1 -- start start 2024-04-27T01:04:57.016 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.179+0000 7f890b3f8700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8904079750 con 0x7f8904106b50 2024-04-27T01:04:57.016 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.179+0000 7f8909194700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f8904106b50 0x7f8904108f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55688/0 (socket says 172.21.15.162:55688) 2024-04-27T01:04:57.016 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.179+0000 7f8909194700 1 -- 172.21.15.162:0/4116292361 learned_addr learned my addr 172.21.15.162:0/4116292361 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:57.016 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.179+0000 7f88fbfff700 1 -- 172.21.15.162:0/4116292361 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948611966 0 0) 0x7f8904079750 con 0x7f8904106b50 2024-04-27T01:04:57.017 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.179+0000 7f88fbfff700 1 -- 172.21.15.162:0/4116292361 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f00035b0 con 0x7f8904106b50 2024-04-27T01:04:57.017 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.180+0000 7f88fbfff700 1 -- 172.21.15.162:0/4116292361 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1482737727 0 0) 0x7f88f00035b0 con 0x7f8904106b50 2024-04-27T01:04:57.017 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.180+0000 7f88fbfff700 1 -- 172.21.15.162:0/4116292361 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890410b670 con 0x7f8904106b50 2024-04-27T01:04:57.017 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.180+0000 7f88fbfff700 1 -- 172.21.15.162:0/4116292361 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f8900001580 con 0x7f8904106b50 2024-04-27T01:04:57.017 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.180+0000 7f88fbfff700 1 -- 172.21.15.162:0/4116292361 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f8900001d20 con 0x7f8904106b50 2024-04-27T01:04:57.017 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.180+0000 7f88fbfff700 1 -- 172.21.15.162:0/4116292361 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f8900005960 con 0x7f8904106b50 2024-04-27T01:04:57.017 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.180+0000 7f890b3f8700 1 -- 172.21.15.162:0/4116292361 >> v1:172.21.15.162:6789/0 conn(0x7f8904106b50 legacy=0x7f8904108f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:57.017 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.180+0000 7f890b3f8700 1 -- 172.21.15.162:0/4116292361 shutdown_connections 2024-04-27T01:04:57.017 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.180+0000 7f890b3f8700 1 -- 172.21.15.162:0/4116292361 >> 172.21.15.162:0/4116292361 conn(0x7f8904100560 msgr2=0x7f89041029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.180+0000 7f890b3f8700 1 -- 172.21.15.162:0/4116292361 shutdown_connections 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.180+0000 7f890b3f8700 1 -- 172.21.15.162:0/4116292361 wait complete. 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.181+0000 7f890b3f8700 1 Processor -- start 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.181+0000 7f890b3f8700 1 -- start start 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.181+0000 7f890b3f8700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f890419fc40 con 0x7f8904106b50 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.181+0000 7f8909194700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f8904106b50 0x7f89041063d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55694/0 (socket says 172.21.15.162:55694) 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.181+0000 7f8909194700 1 -- 172.21.15.162:0/619866156 learned_addr learned my addr 172.21.15.162:0/619866156 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.181+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 792440179 0 0) 0x7f890419fc40 con 0x7f8904106b50 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.181+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f0005530 con 0x7f8904106b50 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.182+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3779568600 0 0) 0x7f88f0005530 con 0x7f8904106b50 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.182+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f890419fc40 con 0x7f8904106b50 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.182+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f8900004870 con 0x7f8904106b50 2024-04-27T01:04:57.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.182+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3537973881 0 0) 0x7f890419fc40 con 0x7f8904106b50 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.182+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89041a0e20 con 0x7f8904106b50 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.182+0000 7f890b3f8700 1 -- 172.21.15.162:0/619866156 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f890419fe70 con 0x7f8904106b50 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.182+0000 7f890b3f8700 1 -- 172.21.15.162:0/619866156 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f89041a03b0 con 0x7f8904106b50 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.182+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f8900004ab0 con 0x7f8904106b50 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.182+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f8900006350 con 0x7f8904106b50 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.183+0000 7f890b3f8700 1 -- 172.21.15.162:0/619866156 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f890410b160 con 0x7f8904106b50 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.183+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 7) ==== 46146+0+0 (unknown 2026620879 0 0) 0x7f8900011940 con 0x7f8904106b50 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.183+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7f89000476a0 con 0x7f8904106b50 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.188+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f8900017550 con 0x7f8904106b50 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.411+0000 7f890b3f8700 1 -- 172.21.15.162:0/619866156 --> v1:172.21.15.162:6800/106205076 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f890402dc40 con 0x7f88f003a230 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.418+0000 7f88fa7fc700 1 -- 172.21.15.162:0/619866156 <== mgr.14120 v1:172.21.15.162:6800/106205076 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f890402dc40 con 0x7f88f003a230 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.421+0000 7f890b3f8700 1 -- 172.21.15.162:0/619866156 >> v1:172.21.15.162:6800/106205076 conn(0x7f88f003a230 legacy=0x7f88f003c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.421+0000 7f890b3f8700 1 -- 172.21.15.162:0/619866156 >> v1:172.21.15.162:6789/0 conn(0x7f8904106b50 legacy=0x7f89041063d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:57.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.421+0000 7f890b3f8700 1 -- 172.21.15.162:0/619866156 shutdown_connections 2024-04-27T01:04:57.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.421+0000 7f890b3f8700 1 -- 172.21.15.162:0/619866156 >> 172.21.15.162:0/619866156 conn(0x7f8904100560 msgr2=0x7f89041908b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:57.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.421+0000 7f890b3f8700 1 -- 172.21.15.162:0/619866156 shutdown_connections 2024-04-27T01:04:57.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:56.421+0000 7f890b3f8700 1 -- 172.21.15.162:0/619866156 wait complete. 2024-04-27T01:04:57.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:57 smithi162 ceph-mon[26714]: [27/Apr/2024:01:04:56] ENGINE Bus STARTING 2024-04-27T01:04:57.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:57 smithi162 ceph-mon[26714]: from='client.14132 v1:172.21.15.162:0/619866156' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:04:57.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:57 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:04:57.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:57 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:57.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:57 smithi162 ceph-mon[26714]: [27/Apr/2024:01:04:56] ENGINE Serving on https://172.21.15.162:7150 2024-04-27T01:04:57.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:57 smithi162 ceph-mon[26714]: [27/Apr/2024:01:04:56] ENGINE Serving on http://172.21.15.162:8765 2024-04-27T01:04:57.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:57 smithi162 ceph-mon[26714]: [27/Apr/2024:01:04:56] ENGINE Bus STARTED 2024-04-27T01:04:57.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:57 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:04:57.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:57 smithi162 ceph-mon[26714]: mgrmap e8: y(active, since 2s) 2024-04-27T01:04:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-27T01:04:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.409+0000 7fedf98d3700 1 Processor -- start 2024-04-27T01:04:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.409+0000 7fedf98d3700 1 -- start start 2024-04-27T01:04:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.409+0000 7fedf98d3700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fedf410a520 con 0x7fedf41060f0 2024-04-27T01:04:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.409+0000 7fedf2ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fedf41060f0 0x7fedf4106510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55710/0 (socket says 172.21.15.162:55710) 2024-04-27T01:04:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.409+0000 7fedf2ffd700 1 -- 172.21.15.162:0/637147555 learned_addr learned my addr 172.21.15.162:0/637147555 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.409+0000 7fedf1ffb700 1 -- 172.21.15.162:0/637147555 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3998077160 0 0) 0x7fedf410a520 con 0x7fedf41060f0 2024-04-27T01:04:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.409+0000 7fedf1ffb700 1 -- 172.21.15.162:0/637147555 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feddc0035b0 con 0x7fedf41060f0 2024-04-27T01:04:58.188 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.409+0000 7fedf1ffb700 1 -- 172.21.15.162:0/637147555 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3289580657 0 0) 0x7feddc0035b0 con 0x7fedf41060f0 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.409+0000 7fedf1ffb700 1 -- 172.21.15.162:0/637147555 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf410b700 con 0x7fedf41060f0 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.409+0000 7fedf1ffb700 1 -- 172.21.15.162:0/637147555 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fede8001580 con 0x7fedf41060f0 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.410+0000 7fedf1ffb700 1 -- 172.21.15.162:0/637147555 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7fede8001c50 con 0x7fedf41060f0 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.410+0000 7fedf1ffb700 1 -- 172.21.15.162:0/637147555 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fede8005960 con 0x7fedf41060f0 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.410+0000 7fedf98d3700 1 -- 172.21.15.162:0/637147555 >> v1:172.21.15.162:6789/0 conn(0x7fedf41060f0 legacy=0x7fedf4106510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.410+0000 7fedf98d3700 1 -- 172.21.15.162:0/637147555 shutdown_connections 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.410+0000 7fedf98d3700 1 -- 172.21.15.162:0/637147555 >> 172.21.15.162:0/637147555 conn(0x7fedf4101670 msgr2=0x7fedf4103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.410+0000 7fedf98d3700 1 -- 172.21.15.162:0/637147555 shutdown_connections 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.410+0000 7fedf98d3700 1 -- 172.21.15.162:0/637147555 wait complete. 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.411+0000 7fedf98d3700 1 Processor -- start 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.411+0000 7fedf98d3700 1 -- start start 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.411+0000 7fedf98d3700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fedf419a3d0 con 0x7fedf41060f0 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.411+0000 7fedf2ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fedf41060f0 0x7fedf4199cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55726/0 (socket says 172.21.15.162:55726) 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.411+0000 7fedf2ffd700 1 -- 172.21.15.162:0/1677409687 learned_addr learned my addr 172.21.15.162:0/1677409687 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:58.189 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.411+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1514195709 0 0) 0x7fedf419a3d0 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.411+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feddc005580 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.412+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3022691454 0 0) 0x7feddc005580 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.412+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedf419a3d0 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.412+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fede8001080 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.412+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2590338334 0 0) 0x7fedf419a3d0 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.412+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf419a5a0 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.412+0000 7fedf98d3700 1 -- 172.21.15.162:0/1677409687 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fedf419a890 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.412+0000 7fedf98d3700 1 -- 172.21.15.162:0/1677409687 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fedf419e3e0 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.412+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7fede80058f0 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.412+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fede8005f30 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.413+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 8) ==== 46252+0+0 (unknown 3073341052 0 0) 0x7fede80071c0 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.413+0000 7fedf98d3700 1 -- 172.21.15.162:0/1677409687 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede0005150 con 0x7fedf41060f0 2024-04-27T01:04:58.190 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.413+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7fede8047770 con 0x7fedf41060f0 2024-04-27T01:04:58.191 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.418+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fede8017470 con 0x7fedf41060f0 2024-04-27T01:04:58.191 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.617+0000 7fedf98d3700 1 -- 172.21.15.162:0/1677409687 --> v1:172.21.15.162:6800/106205076 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fede0000bc0 con 0x7feddc03a4b0 2024-04-27T01:04:58.191 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.617+0000 7fedf88d1700 1 -- 172.21.15.162:0/1677409687 <== mgr.14120 v1:172.21.15.162:6800/106205076 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (unknown 0 0 2070689548) 0x7fede0000bc0 con 0x7feddc03a4b0 2024-04-27T01:04:58.191 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.619+0000 7fedf98d3700 1 -- 172.21.15.162:0/1677409687 >> v1:172.21.15.162:6800/106205076 conn(0x7feddc03a4b0 legacy=0x7feddc03c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:58.191 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.619+0000 7fedf98d3700 1 -- 172.21.15.162:0/1677409687 >> v1:172.21.15.162:6789/0 conn(0x7fedf41060f0 legacy=0x7fedf4199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:58.191 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.620+0000 7fedf98d3700 1 -- 172.21.15.162:0/1677409687 shutdown_connections 2024-04-27T01:04:58.191 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.620+0000 7fedf98d3700 1 -- 172.21.15.162:0/1677409687 >> 172.21.15.162:0/1677409687 conn(0x7fedf4101670 msgr2=0x7fedf4190900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:58.191 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.620+0000 7fedf98d3700 1 -- 172.21.15.162:0/1677409687 shutdown_connections 2024-04-27T01:04:58.191 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:57.620+0000 7fedf98d3700 1 -- 172.21.15.162:0/1677409687 wait complete. 2024-04-27T01:04:58.191 INFO:teuthology.orchestra.run.smithi162.stdout:Generating ssh key... 2024-04-27T01:04:58.467 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-mon[26714]: from='client.14134 v1:172.21.15.162:0/1677409687' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:04:59.092 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: Generating public/private rsa key pair. 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: Your identification has been saved in /tmp/tmpsvyto53s/key. 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: Your public key has been saved in /tmp/tmpsvyto53s/key.pub. 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: The key fingerprint is: 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: SHA256:O33W825+xgml/5LhvLMWyqnOKfL7WgaWQWKacbikC+8 ceph-ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: The key's randomart image is: 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: +---[RSA 3072]----+ 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: | ..+ . | 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: | o* o | 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: | oo. . | 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: |. . . o . | 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: | o . S o | 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: | o . + +o | 2024-04-27T01:04:59.093 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: | . o +.o===.| 2024-04-27T01:04:59.094 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: | E . .= ++ B==| 2024-04-27T01:04:59.094 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: | o+**. .oXB| 2024-04-27T01:04:59.094 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:04:58 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: +----[SHA256]-----+ 2024-04-27T01:04:59.469 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.571+0000 7fee724d3700 1 Processor -- start 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.572+0000 7fee724d3700 1 -- start start 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.572+0000 7fee724d3700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fee6c0796d0 con 0x7fee6c106c00 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.572+0000 7fee6bfff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fee6c106c00 0x7fee6c108ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55740/0 (socket says 172.21.15.162:55740) 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.572+0000 7fee6bfff700 1 -- 172.21.15.162:0/2404397035 learned_addr learned my addr 172.21.15.162:0/2404397035 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.572+0000 7fee6affd700 1 -- 172.21.15.162:0/2404397035 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2124861505 0 0) 0x7fee6c0796d0 con 0x7fee6c106c00 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.572+0000 7fee6affd700 1 -- 172.21.15.162:0/2404397035 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee540035b0 con 0x7fee6c106c00 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.572+0000 7fee6affd700 1 -- 172.21.15.162:0/2404397035 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1509188583 0 0) 0x7fee540035b0 con 0x7fee6c106c00 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.572+0000 7fee6affd700 1 -- 172.21.15.162:0/2404397035 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6c10b720 con 0x7fee6c106c00 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.572+0000 7fee6affd700 1 -- 172.21.15.162:0/2404397035 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fee60001580 con 0x7fee6c106c00 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.573+0000 7fee6affd700 1 -- 172.21.15.162:0/2404397035 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7fee60001d20 con 0x7fee6c106c00 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.573+0000 7fee6affd700 1 -- 172.21.15.162:0/2404397035 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fee60005960 con 0x7fee6c106c00 2024-04-27T01:04:59.470 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.573+0000 7fee724d3700 1 -- 172.21.15.162:0/2404397035 >> v1:172.21.15.162:6789/0 conn(0x7fee6c106c00 legacy=0x7fee6c108ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.573+0000 7fee724d3700 1 -- 172.21.15.162:0/2404397035 shutdown_connections 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.573+0000 7fee724d3700 1 -- 172.21.15.162:0/2404397035 >> 172.21.15.162:0/2404397035 conn(0x7fee6c100590 msgr2=0x7fee6c1029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.573+0000 7fee724d3700 1 -- 172.21.15.162:0/2404397035 shutdown_connections 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.573+0000 7fee724d3700 1 -- 172.21.15.162:0/2404397035 wait complete. 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.574+0000 7fee724d3700 1 Processor -- start 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.574+0000 7fee724d3700 1 -- start start 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.574+0000 7fee724d3700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fee6c19a3c0 con 0x7fee6c106c00 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.574+0000 7fee6bfff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fee6c106c00 0x7fee6c199cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55744/0 (socket says 172.21.15.162:55744) 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.574+0000 7fee6bfff700 1 -- 172.21.15.162:0/2253014468 learned_addr learned my addr 172.21.15.162:0/2253014468 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.574+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2235946600 0 0) 0x7fee6c19a3c0 con 0x7fee6c106c00 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.574+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee54005530 con 0x7fee6c106c00 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.575+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1406018713 0 0) 0x7fee54005530 con 0x7fee6c106c00 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.575+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee6c19a3c0 con 0x7fee6c106c00 2024-04-27T01:04:59.471 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.575+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fee60001080 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.575+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4054001545 0 0) 0x7fee6c19a3c0 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.575+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6c19a590 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.575+0000 7fee724d3700 1 -- 172.21.15.162:0/2253014468 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee6c19a880 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.575+0000 7fee724d3700 1 -- 172.21.15.162:0/2253014468 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fee6c19e3d0 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.575+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7fee60004ab0 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.575+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fee60005f30 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.576+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 8) ==== 46252+0+0 (unknown 3073341052 0 0) 0x7fee60011590 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.576+0000 7fee724d3700 1 -- 172.21.15.162:0/2253014468 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee58005120 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.576+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7fee600477f0 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.581+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fee600174c0 con 0x7fee6c106c00 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.777+0000 7fee724d3700 1 -- 172.21.15.162:0/2253014468 --> v1:172.21.15.162:6800/106205076 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fee58000b90 con 0x7fee5403a4c0 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.890+0000 7fee697fa700 1 -- 172.21.15.162:0/2253014468 <== mgr.14120 v1:172.21.15.162:6800/106205076 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7fee58000b90 con 0x7fee5403a4c0 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.892+0000 7fee724d3700 1 -- 172.21.15.162:0/2253014468 >> v1:172.21.15.162:6800/106205076 conn(0x7fee5403a4c0 legacy=0x7fee5403c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.892+0000 7fee724d3700 1 -- 172.21.15.162:0/2253014468 >> v1:172.21.15.162:6789/0 conn(0x7fee6c106c00 legacy=0x7fee6c199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.892+0000 7fee724d3700 1 -- 172.21.15.162:0/2253014468 shutdown_connections 2024-04-27T01:04:59.472 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.892+0000 7fee724d3700 1 -- 172.21.15.162:0/2253014468 >> 172.21.15.162:0/2253014468 conn(0x7fee6c100590 msgr2=0x7fee6c1908e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:04:59.473 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.892+0000 7fee724d3700 1 -- 172.21.15.162:0/2253014468 shutdown_connections 2024-04-27T01:04:59.473 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:58.892+0000 7fee724d3700 1 -- 172.21.15.162:0/2253014468 wait complete. 2024-04-27T01:05:00.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:59 smithi162 ceph-mon[26714]: from='client.14136 v1:172.21.15.162:0/2253014468' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:00.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:59 smithi162 ceph-mon[26714]: Generating ssh key... 2024-04-27T01:05:00.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:59 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:05:00.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:04:59 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:05:00.693 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllRkhIxUhtQJPpMVc/E/h5zKwOw+MQ6sel0XMkgLcobJp/5fLUSPzqSwnByyfPjMal88vHv9UKkssBdQBdJiMAhuQ+yn3Y4+2pe7B2lLjGjnnfbfCw20OJuNffy3H2i6gp3fZ5uhKH0oQnrnQ/3iBvr0797LzjYNfqc/SwYQbgUmehnA8vAAuTv2aUQEpYbVixCTgRbudUUs565ZnlMr74+ZWGvDlYyEW7PsR5A0k1VCAz7fwvMW0sCt1GMUSArbh6Di9M1Xt6LIbxbnf7Vrq0OrCeLeQmzo+NrKmr5C6iKciVnv8zRr905ASjUjKSjmrW+tXTD/2DnnaF10NmAk5BXkFqa38PRBZnQGOHCFzNRvK8QyupSiGDC4Qa05ywZTQ5/1mDjl20vcK4kuQFSdhpar58FpO8QoVL6XyuHCcfYt0GCWWK5w44nk95sNgOuIzBuxy3h97SMnR6fFf9cJ/oQeeQL2Rny5hgtGYFAK8Sza5iHYW+3chEO8qhyJQfc0= ceph-ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:05:00.693 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.862+0000 7f8749507700 1 Processor -- start 2024-04-27T01:05:00.693 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.862+0000 7f8749507700 1 -- start start 2024-04-27T01:05:00.693 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.863+0000 7f8749507700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8744109410 con 0x7f8744104fe0 2024-04-27T01:05:00.693 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.863+0000 7f8742ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f8744104fe0 0x7f8744105400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55748/0 (socket says 172.21.15.162:55748) 2024-04-27T01:05:00.693 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.863+0000 7f8742ffd700 1 -- 172.21.15.162:0/2324162254 learned_addr learned my addr 172.21.15.162:0/2324162254 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.863+0000 7f8741ffb700 1 -- 172.21.15.162:0/2324162254 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 55665991 0 0) 0x7f8744109410 con 0x7f8744104fe0 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.863+0000 7f8741ffb700 1 -- 172.21.15.162:0/2324162254 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f872c0035b0 con 0x7f8744104fe0 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.863+0000 7f8741ffb700 1 -- 172.21.15.162:0/2324162254 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 505194653 0 0) 0x7f872c0035b0 con 0x7f8744104fe0 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.863+0000 7f8741ffb700 1 -- 172.21.15.162:0/2324162254 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874410a5f0 con 0x7f8744104fe0 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.863+0000 7f8741ffb700 1 -- 172.21.15.162:0/2324162254 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f8738001580 con 0x7f8744104fe0 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.863+0000 7f8741ffb700 1 -- 172.21.15.162:0/2324162254 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f8738001d20 con 0x7f8744104fe0 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.863+0000 7f8741ffb700 1 -- 172.21.15.162:0/2324162254 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f8738005960 con 0x7f8744104fe0 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.864+0000 7f8749507700 1 -- 172.21.15.162:0/2324162254 >> v1:172.21.15.162:6789/0 conn(0x7f8744104fe0 legacy=0x7f8744105400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.864+0000 7f8749507700 1 -- 172.21.15.162:0/2324162254 shutdown_connections 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.864+0000 7f8749507700 1 -- 172.21.15.162:0/2324162254 >> 172.21.15.162:0/2324162254 conn(0x7f8744100560 msgr2=0x7f87441029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.864+0000 7f8749507700 1 -- 172.21.15.162:0/2324162254 shutdown_connections 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.864+0000 7f8749507700 1 -- 172.21.15.162:0/2324162254 wait complete. 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.865+0000 7f8749507700 1 Processor -- start 2024-04-27T01:05:00.694 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.865+0000 7f8749507700 1 -- start start 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.865+0000 7f8749507700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87441991c0 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.865+0000 7f8742ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f8744104fe0 0x7f8744198ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55760/0 (socket says 172.21.15.162:55760) 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.865+0000 7f8742ffd700 1 -- 172.21.15.162:0/405294896 learned_addr learned my addr 172.21.15.162:0/405294896 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.865+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696385550 0 0) 0x7f87441991c0 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.865+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f872c005530 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.865+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2884575301 0 0) 0x7f872c005530 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.865+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87441991c0 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.865+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f8738001080 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.866+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2383725571 0 0) 0x7f87441991c0 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.866+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8744199390 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.866+0000 7f8749507700 1 -- 172.21.15.162:0/405294896 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8744199680 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.866+0000 7f8749507700 1 -- 172.21.15.162:0/405294896 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f874419d1d0 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.866+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f8738001900 con 0x7f8744104fe0 2024-04-27T01:05:00.695 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.866+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f8738006000 con 0x7f8744104fe0 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.867+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 8) ==== 46252+0+0 (unknown 3073341052 0 0) 0x7f8738011740 con 0x7f8744104fe0 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.867+0000 7f8749507700 1 -- 172.21.15.162:0/405294896 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8730005150 con 0x7f8744104fe0 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.867+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7f8738047a20 con 0x7f8744104fe0 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:04:59.872+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f87380176d0 con 0x7f8744104fe0 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:00.068+0000 7f8749507700 1 -- 172.21.15.162:0/405294896 --> v1:172.21.15.162:6800/106205076 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f8730000bc0 con 0x7f872c03a420 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:00.069+0000 7f872bfff700 1 -- 172.21.15.162:0/405294896 <== mgr.14120 v1:172.21.15.162:6800/106205076 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (unknown 0 0 1494626999) 0x7f8730000bc0 con 0x7f872c03a420 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:00.071+0000 7f8749507700 1 -- 172.21.15.162:0/405294896 >> v1:172.21.15.162:6800/106205076 conn(0x7f872c03a420 legacy=0x7f872c03c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:00.071+0000 7f8749507700 1 -- 172.21.15.162:0/405294896 >> v1:172.21.15.162:6789/0 conn(0x7f8744104fe0 legacy=0x7f8744198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:00.071+0000 7f8749507700 1 -- 172.21.15.162:0/405294896 shutdown_connections 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:00.071+0000 7f8749507700 1 -- 172.21.15.162:0/405294896 >> 172.21.15.162:0/405294896 conn(0x7f8744100560 msgr2=0x7f87441029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:00.072+0000 7f8749507700 1 -- 172.21.15.162:0/405294896 shutdown_connections 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:00.072+0000 7f8749507700 1 -- 172.21.15.162:0/405294896 wait complete. 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:Adding key to root@localhost authorized_keys... 2024-04-27T01:05:00.696 INFO:teuthology.orchestra.run.smithi162.stdout:Adding host smithi162... 2024-04-27T01:05:01.299 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:00 smithi162 ceph-mon[26714]: from='client.14138 v1:172.21.15.162:0/405294896' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:02.158 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:01 smithi162 ceph-mon[26714]: from='client.14140 v1:172.21.15.162:0/2248491062' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "addr": "172.21.15.162", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:03.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:02 smithi162 ceph-mon[26714]: Deploying cephadm binary to smithi162 2024-04-27T01:05:05.927 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout Added host 'smithi162' with addr '172.21.15.162' 2024-04-27T01:05:05.927 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.096+0000 7fe78a77a700 1 Processor -- start 2024-04-27T01:05:05.927 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.096+0000 7fe78a77a700 1 -- start start 2024-04-27T01:05:05.927 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.096+0000 7fe78a77a700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe78410a520 con 0x7fe7841060f0 2024-04-27T01:05:05.927 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.096+0000 7fe783fff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe7841060f0 0x7fe784106510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55764/0 (socket says 172.21.15.162:55764) 2024-04-27T01:05:05.927 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.097+0000 7fe783fff700 1 -- 172.21.15.162:0/1140163353 learned_addr learned my addr 172.21.15.162:0/1140163353 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:05.927 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.097+0000 7fe782ffd700 1 -- 172.21.15.162:0/1140163353 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2777095302 0 0) 0x7fe78410a520 con 0x7fe7841060f0 2024-04-27T01:05:05.927 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.097+0000 7fe782ffd700 1 -- 172.21.15.162:0/1140163353 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe76c0035b0 con 0x7fe7841060f0 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.097+0000 7fe782ffd700 1 -- 172.21.15.162:0/1140163353 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3696241598 0 0) 0x7fe76c0035b0 con 0x7fe7841060f0 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.097+0000 7fe782ffd700 1 -- 172.21.15.162:0/1140163353 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78410b700 con 0x7fe7841060f0 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.097+0000 7fe782ffd700 1 -- 172.21.15.162:0/1140163353 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fe774001580 con 0x7fe7841060f0 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.097+0000 7fe782ffd700 1 -- 172.21.15.162:0/1140163353 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7fe774001d20 con 0x7fe7841060f0 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.097+0000 7fe782ffd700 1 -- 172.21.15.162:0/1140163353 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fe774005960 con 0x7fe7841060f0 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.098+0000 7fe78a77a700 1 -- 172.21.15.162:0/1140163353 >> v1:172.21.15.162:6789/0 conn(0x7fe7841060f0 legacy=0x7fe784106510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.098+0000 7fe78a77a700 1 -- 172.21.15.162:0/1140163353 shutdown_connections 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.098+0000 7fe78a77a700 1 -- 172.21.15.162:0/1140163353 >> 172.21.15.162:0/1140163353 conn(0x7fe784101670 msgr2=0x7fe784103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.098+0000 7fe78a77a700 1 -- 172.21.15.162:0/1140163353 shutdown_connections 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.098+0000 7fe78a77a700 1 -- 172.21.15.162:0/1140163353 wait complete. 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.098+0000 7fe78a77a700 1 Processor -- start 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.098+0000 7fe78a77a700 1 -- start start 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe78a77a700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe78419a340 con 0x7fe7841060f0 2024-04-27T01:05:05.928 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe783fff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe7841060f0 0x7fe784199c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55776/0 (socket says 172.21.15.162:55776) 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe783fff700 1 -- 172.21.15.162:0/2248491062 learned_addr learned my addr 172.21.15.162:0/2248491062 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2421343560 0 0) 0x7fe78419a340 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe76c005530 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4286547584 0 0) 0x7fe76c005530 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe78419a340 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fe774004870 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1262224050 0 0) 0x7fe78419a340 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78419a510 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe78a77a700 1 -- 172.21.15.162:0/2248491062 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe78419a800 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.099+0000 7fe78a77a700 1 -- 172.21.15.162:0/2248491062 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe78419e350 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.100+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7fe774001d20 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.100+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fe774005f60 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.100+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 8) ==== 46252+0+0 (unknown 3073341052 0 0) 0x7fe7740115e0 con 0x7fe7841060f0 2024-04-27T01:05:05.929 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.101+0000 7fe78a77a700 1 -- 172.21.15.162:0/2248491062 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe770005150 con 0x7fe7841060f0 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.101+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7fe774047910 con 0x7fe7841060f0 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.105+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fe774017530 con 0x7fe7841060f0 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:01.306+0000 7fe78a77a700 1 -- 172.21.15.162:0/2248491062 --> v1:172.21.15.162:6800/106205076 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi162", "addr": "172.21.15.162", "target": ["mon-mgr", ""]}) -- 0x7fe770000bc0 con 0x7fe76c03a2d0 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:05.352+0000 7fe7817fa700 1 -- 172.21.15.162:0/2248491062 <== mgr.14120 v1:172.21.15.162:6800/106205076 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (unknown 0 0 2577743529) 0x7fe770000bc0 con 0x7fe76c03a2d0 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:05.355+0000 7fe78a77a700 1 -- 172.21.15.162:0/2248491062 >> v1:172.21.15.162:6800/106205076 conn(0x7fe76c03a2d0 legacy=0x7fe76c03c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:05.355+0000 7fe78a77a700 1 -- 172.21.15.162:0/2248491062 >> v1:172.21.15.162:6789/0 conn(0x7fe7841060f0 legacy=0x7fe784199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:05.355+0000 7fe78a77a700 1 -- 172.21.15.162:0/2248491062 shutdown_connections 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:05.355+0000 7fe78a77a700 1 -- 172.21.15.162:0/2248491062 >> 172.21.15.162:0/2248491062 conn(0x7fe784101670 msgr2=0x7fe784103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:05.355+0000 7fe78a77a700 1 -- 172.21.15.162:0/2248491062 shutdown_connections 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:05.355+0000 7fe78a77a700 1 -- 172.21.15.162:0/2248491062 wait complete. 2024-04-27T01:05:05.930 INFO:teuthology.orchestra.run.smithi162.stdout:Deploying unmanaged mon service... 2024-04-27T01:05:06.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:06 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:05:06.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:06 smithi162 ceph-mon[26714]: Added host smithi162 2024-04-27T01:05:06.450 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:06 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:05:07.331 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-27T01:05:07.331 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.410+0000 7f515f3dc700 1 Processor -- start 2024-04-27T01:05:07.331 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.410+0000 7f515f3dc700 1 -- start start 2024-04-27T01:05:07.331 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.410+0000 7f515f3dc700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5158079710 con 0x7f5158106b90 2024-04-27T01:05:07.331 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.410+0000 7f515d178700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f5158106b90 0x7f5158108f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38054/0 (socket says 172.21.15.162:38054) 2024-04-27T01:05:07.331 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.410+0000 7f515d178700 1 -- 172.21.15.162:0/286356939 learned_addr learned my addr 172.21.15.162:0/286356939 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:07.331 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.410+0000 7f514ffff700 1 -- 172.21.15.162:0/286356939 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240187082 0 0) 0x7f5158079710 con 0x7f5158106b90 2024-04-27T01:05:07.331 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.410+0000 7f514ffff700 1 -- 172.21.15.162:0/286356939 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51440035b0 con 0x7f5158106b90 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.411+0000 7f514ffff700 1 -- 172.21.15.162:0/286356939 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1271774675 0 0) 0x7f51440035b0 con 0x7f5158106b90 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.411+0000 7f514ffff700 1 -- 172.21.15.162:0/286356939 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515810b6b0 con 0x7f5158106b90 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.411+0000 7f514ffff700 1 -- 172.21.15.162:0/286356939 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f5148001580 con 0x7f5158106b90 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.411+0000 7f514ffff700 1 -- 172.21.15.162:0/286356939 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f5148001c50 con 0x7f5158106b90 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.411+0000 7f514ffff700 1 -- 172.21.15.162:0/286356939 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f5148005960 con 0x7f5158106b90 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.411+0000 7f515f3dc700 1 -- 172.21.15.162:0/286356939 >> v1:172.21.15.162:6789/0 conn(0x7f5158106b90 legacy=0x7f5158108f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.411+0000 7f515f3dc700 1 -- 172.21.15.162:0/286356939 shutdown_connections 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.411+0000 7f515f3dc700 1 -- 172.21.15.162:0/286356939 >> 172.21.15.162:0/286356939 conn(0x7f5158100560 msgr2=0x7f51581029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.411+0000 7f515f3dc700 1 -- 172.21.15.162:0/286356939 shutdown_connections 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.411+0000 7f515f3dc700 1 -- 172.21.15.162:0/286356939 wait complete. 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.412+0000 7f515f3dc700 1 Processor -- start 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.412+0000 7f515f3dc700 1 -- start start 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.412+0000 7f515f3dc700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f515819c590 con 0x7f5158106b90 2024-04-27T01:05:07.332 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.412+0000 7f515d178700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f5158106b90 0x7f515819be80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38070/0 (socket says 172.21.15.162:38070) 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.412+0000 7f515d178700 1 -- 172.21.15.162:0/859580061 learned_addr learned my addr 172.21.15.162:0/859580061 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.412+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2156826470 0 0) 0x7f515819c590 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.412+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5144005580 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.412+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2901474352 0 0) 0x7f5144005580 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.412+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f515819c590 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.412+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f5148001080 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.413+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4108339046 0 0) 0x7f515819c590 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.413+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515819c760 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.413+0000 7f515f3dc700 1 -- 172.21.15.162:0/859580061 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f515819ca50 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.413+0000 7f515f3dc700 1 -- 172.21.15.162:0/859580061 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f51581a05a0 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.413+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1037+0+0 (unknown 3899978296 0 0) 0x7f51480017f0 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.413+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f5148006100 con 0x7f5158106b90 2024-04-27T01:05:07.333 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.413+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 8) ==== 46252+0+0 (unknown 3073341052 0 0) 0x7f5148011760 con 0x7f5158106b90 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.413+0000 7f515f3dc700 1 -- 172.21.15.162:0/859580061 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f513c005120 con 0x7f5158106b90 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.414+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7f5148047950 con 0x7f5158106b90 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.419+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f5148017640 con 0x7f5158106b90 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.646+0000 7f515f3dc700 1 -- 172.21.15.162:0/859580061 --> v1:172.21.15.162:6800/106205076 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f513c000b90 con 0x7f514403a460 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.653+0000 7f514e7fc700 1 -- 172.21.15.162:0/859580061 <== mgr.14120 v1:172.21.15.162:6800/106205076 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 3265049985) 0x7f513c000b90 con 0x7f514403a460 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.655+0000 7f515f3dc700 1 -- 172.21.15.162:0/859580061 >> v1:172.21.15.162:6800/106205076 conn(0x7f514403a460 legacy=0x7f514403c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.655+0000 7f515f3dc700 1 -- 172.21.15.162:0/859580061 >> v1:172.21.15.162:6789/0 conn(0x7f5158106b90 legacy=0x7f515819be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.655+0000 7f515f3dc700 1 -- 172.21.15.162:0/859580061 shutdown_connections 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.655+0000 7f515f3dc700 1 -- 172.21.15.162:0/859580061 >> 172.21.15.162:0/859580061 conn(0x7f5158100560 msgr2=0x7f51581049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.655+0000 7f515f3dc700 1 -- 172.21.15.162:0/859580061 shutdown_connections 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:06.655+0000 7f515f3dc700 1 -- 172.21.15.162:0/859580061 wait complete. 2024-04-27T01:05:07.334 INFO:teuthology.orchestra.run.smithi162.stdout:Deploying unmanaged mgr service... 2024-04-27T01:05:08.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:07 smithi162 ceph-mon[26714]: from='client.14142 v1:172.21.15.162:0/859580061' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:08.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:07 smithi162 ceph-mon[26714]: Saving service mon spec with placement count:5 2024-04-27T01:05:08.070 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:07 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:05:09.342 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-27T01:05:09.342 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.541+0000 7efcd590b700 1 Processor -- start 2024-04-27T01:05:09.342 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.541+0000 7efcd590b700 1 -- start start 2024-04-27T01:05:09.342 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.541+0000 7efcd590b700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efcd00796f0 con 0x7efcd0106be0 2024-04-27T01:05:09.342 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.542+0000 7efcceffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7efcd0106be0 0x7efcd0108fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38082/0 (socket says 172.21.15.162:38082) 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.542+0000 7efcceffd700 1 -- 172.21.15.162:0/2475572580 learned_addr learned my addr 172.21.15.162:0/2475572580 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.542+0000 7efccdffb700 1 -- 172.21.15.162:0/2475572580 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3012455924 0 0) 0x7efcd00796f0 con 0x7efcd0106be0 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.542+0000 7efccdffb700 1 -- 172.21.15.162:0/2475572580 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcb80035b0 con 0x7efcd0106be0 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.542+0000 7efccdffb700 1 -- 172.21.15.162:0/2475572580 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3961717690 0 0) 0x7efcb80035b0 con 0x7efcd0106be0 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.542+0000 7efccdffb700 1 -- 172.21.15.162:0/2475572580 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd010b700 con 0x7efcd0106be0 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.542+0000 7efccdffb700 1 -- 172.21.15.162:0/2475572580 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7efcc4001580 con 0x7efcd0106be0 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.542+0000 7efccdffb700 1 -- 172.21.15.162:0/2475572580 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7efcc4004930 con 0x7efcd0106be0 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.542+0000 7efccdffb700 1 -- 172.21.15.162:0/2475572580 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7efcc4005a70 con 0x7efcd0106be0 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.543+0000 7efcd590b700 1 -- 172.21.15.162:0/2475572580 >> v1:172.21.15.162:6789/0 conn(0x7efcd0106be0 legacy=0x7efcd0108fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.543+0000 7efcd590b700 1 -- 172.21.15.162:0/2475572580 shutdown_connections 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.543+0000 7efcd590b700 1 -- 172.21.15.162:0/2475572580 >> 172.21.15.162:0/2475572580 conn(0x7efcd0100590 msgr2=0x7efcd01029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.543+0000 7efcd590b700 1 -- 172.21.15.162:0/2475572580 shutdown_connections 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.543+0000 7efcd590b700 1 -- 172.21.15.162:0/2475572580 wait complete. 2024-04-27T01:05:09.343 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.544+0000 7efcd590b700 1 Processor -- start 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.544+0000 7efcd590b700 1 -- start start 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.544+0000 7efcd590b700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7efcd019a3e0 con 0x7efcd0106be0 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.544+0000 7efcceffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7efcd0106be0 0x7efcd0199cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38086/0 (socket says 172.21.15.162:38086) 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.544+0000 7efcceffd700 1 -- 172.21.15.162:0/3519408775 learned_addr learned my addr 172.21.15.162:0/3519408775 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.544+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3898849001 0 0) 0x7efcd019a3e0 con 0x7efcd0106be0 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.544+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcb8005530 con 0x7efcd0106be0 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.544+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4140249553 0 0) 0x7efcb8005530 con 0x7efcd0106be0 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.544+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efcd019a3e0 con 0x7efcd0106be0 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.544+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7efcc4001080 con 0x7efcd0106be0 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.545+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3666728658 0 0) 0x7efcd019a3e0 con 0x7efcd0106be0 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.545+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd019a5b0 con 0x7efcd0106be0 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.545+0000 7efcd590b700 1 -- 172.21.15.162:0/3519408775 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efcd019a8a0 con 0x7efcd0106be0 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.545+0000 7efcd590b700 1 -- 172.21.15.162:0/3519408775 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efcd019e3e0 con 0x7efcd0106be0 2024-04-27T01:05:09.344 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.545+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7efcc4004930 con 0x7efcd0106be0 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.545+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7efcc40060c0 con 0x7efcd0106be0 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.546+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 8) ==== 46252+0+0 (unknown 3073341052 0 0) 0x7efcc4011720 con 0x7efcd0106be0 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.546+0000 7efcd590b700 1 -- 172.21.15.162:0/3519408775 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcbc005120 con 0x7efcd0106be0 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.546+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7efcc4047a90 con 0x7efcd0106be0 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.551+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7efcc4017630 con 0x7efcd0106be0 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.784+0000 7efcd590b700 1 -- 172.21.15.162:0/3519408775 --> v1:172.21.15.162:6800/106205076 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7efcbc000b90 con 0x7efcb803a470 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.791+0000 7efcd4909700 1 -- 172.21.15.162:0/3519408775 <== mgr.14120 v1:172.21.15.162:6800/106205076 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 325935098) 0x7efcbc000b90 con 0x7efcb803a470 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.793+0000 7efcd590b700 1 -- 172.21.15.162:0/3519408775 >> v1:172.21.15.162:6800/106205076 conn(0x7efcb803a470 legacy=0x7efcb803c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.793+0000 7efcd590b700 1 -- 172.21.15.162:0/3519408775 >> v1:172.21.15.162:6789/0 conn(0x7efcd0106be0 legacy=0x7efcd0199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.793+0000 7efcd590b700 1 -- 172.21.15.162:0/3519408775 shutdown_connections 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.793+0000 7efcd590b700 1 -- 172.21.15.162:0/3519408775 >> 172.21.15.162:0/3519408775 conn(0x7efcd0100590 msgr2=0x7efcd0190920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.794+0000 7efcd590b700 1 -- 172.21.15.162:0/3519408775 shutdown_connections 2024-04-27T01:05:09.345 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:08.794+0000 7efcd590b700 1 -- 172.21.15.162:0/3519408775 wait complete. 2024-04-27T01:05:09.461 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:09 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:05:09.461 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:09 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:05:09.461 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:09 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:05:10.288 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:10 smithi162 ceph-mon[26714]: from='client.14144 v1:172.21.15.162:0/3519408775' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:10.288 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:10 smithi162 ceph-mon[26714]: Saving service mgr spec with placement count:2 2024-04-27T01:05:10.288 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/251865230' entity='client.admin' 2024-04-27T01:05:11.096 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.766+0000 7f57185a1700 1 Processor -- start 2024-04-27T01:05:11.096 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.766+0000 7f57185a1700 1 -- start start 2024-04-27T01:05:11.096 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.767+0000 7f57185a1700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f571010a500 con 0x7f57101060d0 2024-04-27T01:05:11.096 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.767+0000 7f571633d700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f57101060d0 0x7f57101064f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38088/0 (socket says 172.21.15.162:38088) 2024-04-27T01:05:11.096 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.767+0000 7f571633d700 1 -- 172.21.15.162:0/371760994 learned_addr learned my addr 172.21.15.162:0/371760994 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:11.097 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.767+0000 7f571533b700 1 -- 172.21.15.162:0/371760994 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710764539 0 0) 0x7f571010a500 con 0x7f57101060d0 2024-04-27T01:05:11.097 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.767+0000 7f571533b700 1 -- 172.21.15.162:0/371760994 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56fc0035b0 con 0x7f57101060d0 2024-04-27T01:05:11.097 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.767+0000 7f571533b700 1 -- 172.21.15.162:0/371760994 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2203994824 0 0) 0x7f56fc0035b0 con 0x7f57101060d0 2024-04-27T01:05:11.097 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.767+0000 7f571533b700 1 -- 172.21.15.162:0/371760994 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571010b6e0 con 0x7f57101060d0 2024-04-27T01:05:11.097 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.767+0000 7f571533b700 1 -- 172.21.15.162:0/371760994 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f570c001900 con 0x7f57101060d0 2024-04-27T01:05:11.097 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.767+0000 7f571533b700 1 -- 172.21.15.162:0/371760994 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f570c004cb0 con 0x7f57101060d0 2024-04-27T01:05:11.097 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.767+0000 7f571533b700 1 -- 172.21.15.162:0/371760994 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f570c005ce0 con 0x7f57101060d0 2024-04-27T01:05:11.097 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.768+0000 7f57185a1700 1 -- 172.21.15.162:0/371760994 >> v1:172.21.15.162:6789/0 conn(0x7f57101060d0 legacy=0x7f57101064f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:11.097 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.768+0000 7f57185a1700 1 -- 172.21.15.162:0/371760994 shutdown_connections 2024-04-27T01:05:11.097 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.768+0000 7f57185a1700 1 -- 172.21.15.162:0/371760994 >> 172.21.15.162:0/371760994 conn(0x7f5710101630 msgr2=0x7f5710103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:11.098 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.768+0000 7f57185a1700 1 -- 172.21.15.162:0/371760994 shutdown_connections 2024-04-27T01:05:11.098 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.768+0000 7f57185a1700 1 -- 172.21.15.162:0/371760994 wait complete. 2024-04-27T01:05:11.098 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57185a1700 1 Processor -- start 2024-04-27T01:05:11.098 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57185a1700 1 -- start start 2024-04-27T01:05:11.098 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57185a1700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f571019a320 con 0x7f57101060d0 2024-04-27T01:05:11.098 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f571633d700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f57101060d0 0x7f5710199c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38102/0 (socket says 172.21.15.162:38102) 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f571633d700 1 -- 172.21.15.162:0/251865230 learned_addr learned my addr 172.21.15.162:0/251865230 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1284136290 0 0) 0x7f571019a320 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56fc005530 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1610579872 0 0) 0x7f56fc005530 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f571019a320 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f570c001400 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3798849906 0 0) 0x7f571019a320 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571019a4f0 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.769+0000 7f57185a1700 1 -- 172.21.15.162:0/251865230 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f571019a7e0 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.770+0000 7f57185a1700 1 -- 172.21.15.162:0/251865230 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f571019e330 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.770+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f570c005d50 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.770+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f570c0066c0 con 0x7f57101060d0 2024-04-27T01:05:11.099 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.770+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 8) ==== 46252+0+0 (unknown 3073341052 0 0) 0x7f570c011d40 con 0x7f57101060d0 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.770+0000 7f57185a1700 1 -- 172.21.15.162:0/251865230 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f4005150 con 0x7f57101060d0 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.771+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7f570c047db0 con 0x7f57101060d0 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.775+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f570c0179a0 con 0x7f57101060d0 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.987+0000 7f57185a1700 1 -- 172.21.15.162:0/251865230 --> v1:172.21.15.162:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f56f4005440 con 0x7f57101060d0 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.990+0000 7f57077fe700 1 -- 172.21.15.162:0/251865230 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (unknown 2684808024 0 0) 0x7f570c0172a0 con 0x7f57101060d0 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.992+0000 7f57185a1700 1 -- 172.21.15.162:0/251865230 >> v1:172.21.15.162:6800/106205076 conn(0x7f56fc03a4c0 legacy=0x7f56fc03c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.992+0000 7f57185a1700 1 -- 172.21.15.162:0/251865230 >> v1:172.21.15.162:6789/0 conn(0x7f57101060d0 legacy=0x7f5710199c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.992+0000 7f57185a1700 1 -- 172.21.15.162:0/251865230 shutdown_connections 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.992+0000 7f57185a1700 1 -- 172.21.15.162:0/251865230 >> 172.21.15.162:0/251865230 conn(0x7f5710101630 msgr2=0x7f5710103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.992+0000 7f57185a1700 1 -- 172.21.15.162:0/251865230 shutdown_connections 2024-04-27T01:05:11.100 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:09.992+0000 7f57185a1700 1 -- 172.21.15.162:0/251865230 wait complete. 2024-04-27T01:05:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:11 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:05:12.080 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2576499659' entity='client.admin' 2024-04-27T01:05:12.370 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.537+0000 7f268c48c700 1 Processor -- start 2024-04-27T01:05:12.370 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.537+0000 7f268c48c700 1 -- start start 2024-04-27T01:05:12.370 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.537+0000 7f268c48c700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2684108b50 con 0x7f2684104720 2024-04-27T01:05:12.370 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.537+0000 7f268a228700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f2684104720 0x7f2684104b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38110/0 (socket says 172.21.15.162:38110) 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.537+0000 7f268a228700 1 -- 172.21.15.162:0/2586208165 learned_addr learned my addr 172.21.15.162:0/2586208165 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.538+0000 7f2689226700 1 -- 172.21.15.162:0/2586208165 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 606634099 0 0) 0x7f2684108b50 con 0x7f2684104720 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.538+0000 7f2689226700 1 -- 172.21.15.162:0/2586208165 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f266c0035b0 con 0x7f2684104720 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.538+0000 7f2689226700 1 -- 172.21.15.162:0/2586208165 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 771886191 0 0) 0x7f266c0035b0 con 0x7f2684104720 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.538+0000 7f2689226700 1 -- 172.21.15.162:0/2586208165 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2684109d30 con 0x7f2684104720 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.538+0000 7f2689226700 1 -- 172.21.15.162:0/2586208165 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f267c001900 con 0x7f2684104720 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.538+0000 7f2689226700 1 -- 172.21.15.162:0/2586208165 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f267c004cb0 con 0x7f2684104720 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.538+0000 7f2689226700 1 -- 172.21.15.162:0/2586208165 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f267c005ce0 con 0x7f2684104720 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.539+0000 7f268c48c700 1 -- 172.21.15.162:0/2586208165 >> v1:172.21.15.162:6789/0 conn(0x7f2684104720 legacy=0x7f2684104b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.539+0000 7f268c48c700 1 -- 172.21.15.162:0/2586208165 shutdown_connections 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.539+0000 7f268c48c700 1 -- 172.21.15.162:0/2586208165 >> 172.21.15.162:0/2586208165 conn(0x7f26840ffca0 msgr2=0x7f2684102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.539+0000 7f268c48c700 1 -- 172.21.15.162:0/2586208165 shutdown_connections 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.539+0000 7f268c48c700 1 -- 172.21.15.162:0/2586208165 wait complete. 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.539+0000 7f268c48c700 1 Processor -- start 2024-04-27T01:05:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.539+0000 7f268c48c700 1 -- start start 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f268c48c700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2684100b70 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f268a228700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f2684104720 0x7f268419c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38126/0 (socket says 172.21.15.162:38126) 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f268a228700 1 -- 172.21.15.162:0/2576499659 learned_addr learned my addr 172.21.15.162:0/2576499659 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2000711569 0 0) 0x7f2684100b70 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f266c005530 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2928548655 0 0) 0x7f266c005530 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2684100b70 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f267c001400 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 585226403 0 0) 0x7f2684100b70 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2684100d40 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f268c48c700 1 -- 172.21.15.162:0/2576499659 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2684101030 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.540+0000 7f268c48c700 1 -- 172.21.15.162:0/2576499659 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f268419e3a0 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.541+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f267c004cb0 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.541+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f267c006340 con 0x7f2684104720 2024-04-27T01:05:12.372 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.541+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 8) ==== 46252+0+0 (unknown 3073341052 0 0) 0x7f267c011a60 con 0x7f2684104720 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.541+0000 7f268c48c700 1 -- 172.21.15.162:0/2576499659 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2670005120 con 0x7f2684104720 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.542+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7f267c047d50 con 0x7f2684104720 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.546+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f267c017980 con 0x7f2684104720 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.756+0000 7f268c48c700 1 -- 172.21.15.162:0/2576499659 --> v1:172.21.15.162:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f2670005410 con 0x7f2684104720 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.759+0000 7f26837fe700 1 -- 172.21.15.162:0/2576499659 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (unknown 1496729260 0 0) 0x7f267c017280 con 0x7f2684104720 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.761+0000 7f268c48c700 1 -- 172.21.15.162:0/2576499659 >> v1:172.21.15.162:6800/106205076 conn(0x7f266c03a470 legacy=0x7f266c03c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.761+0000 7f268c48c700 1 -- 172.21.15.162:0/2576499659 >> v1:172.21.15.162:6789/0 conn(0x7f2684104720 legacy=0x7f268419c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.762+0000 7f268c48c700 1 -- 172.21.15.162:0/2576499659 shutdown_connections 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.762+0000 7f268c48c700 1 -- 172.21.15.162:0/2576499659 >> 172.21.15.162:0/2576499659 conn(0x7f26840ffca0 msgr2=0x7f26841081f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.762+0000 7f268c48c700 1 -- 172.21.15.162:0/2576499659 shutdown_connections 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:11.762+0000 7f268c48c700 1 -- 172.21.15.162:0/2576499659 wait complete. 2024-04-27T01:05:12.373 INFO:teuthology.orchestra.run.smithi162.stdout:Enabling the dashboard module... 2024-04-27T01:05:13.269 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:13 smithi162 ceph-mon[26714]: from='mgr.14120 v1:172.21.15.162:0/3788963284' entity='mgr.y' 2024-04-27T01:05:14.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1205332060' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-27T01:05:14.342 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:14 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ignoring --setuser ceph since I am not root 2024-04-27T01:05:14.343 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:14 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ignoring --setgroup ceph since I am not root 2024-04-27T01:05:14.343 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:14 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:14.235+0000 7ff34da6b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-27T01:05:14.618 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:14 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:14.420+0000 7ff34da6b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:05:14.618 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:14 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:14.521+0000 7ff34da6b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-27T01:05:14.624 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.805+0000 7f99fe77e700 1 Processor -- start 2024-04-27T01:05:14.624 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.805+0000 7f99fe77e700 1 -- start start 2024-04-27T01:05:14.624 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.805+0000 7f99fe77e700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99f810a510 con 0x7f99f81060e0 2024-04-27T01:05:14.624 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.806+0000 7f99f7fff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f99f81060e0 0x7f99f8106500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38128/0 (socket says 172.21.15.162:38128) 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.806+0000 7f99f7fff700 1 -- 172.21.15.162:0/4201848757 learned_addr learned my addr 172.21.15.162:0/4201848757 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.806+0000 7f99f6ffd700 1 -- 172.21.15.162:0/4201848757 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2630538047 0 0) 0x7f99f810a510 con 0x7f99f81060e0 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.806+0000 7f99f6ffd700 1 -- 172.21.15.162:0/4201848757 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99e00035b0 con 0x7f99f81060e0 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.806+0000 7f99f6ffd700 1 -- 172.21.15.162:0/4201848757 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4127135790 0 0) 0x7f99e00035b0 con 0x7f99f81060e0 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.806+0000 7f99f6ffd700 1 -- 172.21.15.162:0/4201848757 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f810b6f0 con 0x7f99f81060e0 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.806+0000 7f99f6ffd700 1 -- 172.21.15.162:0/4201848757 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f99e8001580 con 0x7f99f81060e0 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.806+0000 7f99f6ffd700 1 -- 172.21.15.162:0/4201848757 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f99e8004930 con 0x7f99f81060e0 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.806+0000 7f99f6ffd700 1 -- 172.21.15.162:0/4201848757 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f99e8005960 con 0x7f99f81060e0 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.807+0000 7f99fe77e700 1 -- 172.21.15.162:0/4201848757 >> v1:172.21.15.162:6789/0 conn(0x7f99f81060e0 legacy=0x7f99f8106500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.807+0000 7f99fe77e700 1 -- 172.21.15.162:0/4201848757 shutdown_connections 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.807+0000 7f99fe77e700 1 -- 172.21.15.162:0/4201848757 >> 172.21.15.162:0/4201848757 conn(0x7f99f8101640 msgr2=0x7f99f8103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.807+0000 7f99fe77e700 1 -- 172.21.15.162:0/4201848757 shutdown_connections 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.807+0000 7f99fe77e700 1 -- 172.21.15.162:0/4201848757 wait complete. 2024-04-27T01:05:14.625 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.808+0000 7f99fe77e700 1 Processor -- start 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.808+0000 7f99fe77e700 1 -- start start 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.808+0000 7f99fe77e700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99f819a390 con 0x7f99f81060e0 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.808+0000 7f99f7fff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f99f81060e0 0x7f99f8199c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38138/0 (socket says 172.21.15.162:38138) 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.808+0000 7f99f7fff700 1 -- 172.21.15.162:0/1205332060 learned_addr learned my addr 172.21.15.162:0/1205332060 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.808+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831727005 0 0) 0x7f99f819a390 con 0x7f99f81060e0 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.808+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99e0005530 con 0x7f99f81060e0 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.808+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2598255359 0 0) 0x7f99e0005530 con 0x7f99f81060e0 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.808+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99f819a390 con 0x7f99f81060e0 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.808+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f99e8001080 con 0x7f99f81060e0 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.809+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2473098283 0 0) 0x7f99f819a390 con 0x7f99f81060e0 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.809+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f819a560 con 0x7f99f81060e0 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.809+0000 7f99fe77e700 1 -- 172.21.15.162:0/1205332060 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f99f819a850 con 0x7f99f81060e0 2024-04-27T01:05:14.626 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.809+0000 7f99fe77e700 1 -- 172.21.15.162:0/1205332060 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f99f819e3a0 con 0x7f99f81060e0 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.809+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f99e8004c10 con 0x7f99f81060e0 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.809+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f99e8006220 con 0x7f99f81060e0 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.809+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 8) ==== 46252+0+0 (unknown 3073341052 0 0) 0x7f99e80118a0 con 0x7f99f81060e0 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.810+0000 7f99fe77e700 1 -- 172.21.15.162:0/1205332060 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99f8063890 con 0x7f99f81060e0 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.810+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7f99e8047960 con 0x7f99f81060e0 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:12.815+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f99e8017500 con 0x7f99f81060e0 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:13.083+0000 7f99fe77e700 1 -- 172.21.15.162:0/1205332060 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f99f819e930 con 0x7f99f81060e0 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:14.053+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (unknown 1498667528 0 0) 0x7f99e8016e00 con 0x7f99f81060e0 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:14.054+0000 7f99f57fa700 1 -- 172.21.15.162:0/1205332060 <== mon.0 v1:172.21.15.162:6789/0 11 ==== mgrmap(e 9) ==== 46265+0+0 (unknown 3224811188 0 0) 0x7f99e80462a0 con 0x7f99f81060e0 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:14.057+0000 7f99fe77e700 1 -- 172.21.15.162:0/1205332060 >> v1:172.21.15.162:6800/106205076 conn(0x7f99e003a590 legacy=0x7f99e003ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:14.057+0000 7f99fe77e700 1 -- 172.21.15.162:0/1205332060 >> v1:172.21.15.162:6789/0 conn(0x7f99f81060e0 legacy=0x7f99f8199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:14.057+0000 7f99fe77e700 1 -- 172.21.15.162:0/1205332060 shutdown_connections 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:14.057+0000 7f99fe77e700 1 -- 172.21.15.162:0/1205332060 >> 172.21.15.162:0/1205332060 conn(0x7f99f8101640 msgr2=0x7f99f8103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:14.057+0000 7f99fe77e700 1 -- 172.21.15.162:0/1205332060 shutdown_connections 2024-04-27T01:05:14.627 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:14.057+0000 7f99fe77e700 1 -- 172.21.15.162:0/1205332060 wait complete. 2024-04-27T01:05:14.921 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:14 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:14.837+0000 7ff34da6b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-27T01:05:14.921 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:14 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:14.926+0000 7ff34da6b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-27T01:05:15.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1205332060' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-27T01:05:15.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:15 smithi162 ceph-mon[26714]: mgrmap e9: y(active, since 19s) 2024-04-27T01:05:15.893 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:05:15.893 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-27T01:05:15.893 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-27T01:05:15.893 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-27T01:05:15.893 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-27T01:05:15.893 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:05:15.893 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.046+0000 7f7915d12700 1 Processor -- start 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.047+0000 7f7915d12700 1 -- start start 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.047+0000 7f7915d12700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f791010a500 con 0x7f79101060d0 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.047+0000 7f790f7fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f79101060d0 0x7f79101064f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39738/0 (socket says 172.21.15.162:39738) 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.047+0000 7f790f7fe700 1 -- 172.21.15.162:0/2837159592 learned_addr learned my addr 172.21.15.162:0/2837159592 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.047+0000 7f790e7fc700 1 -- 172.21.15.162:0/2837159592 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2905851089 0 0) 0x7f791010a500 con 0x7f79101060d0 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.047+0000 7f790e7fc700 1 -- 172.21.15.162:0/2837159592 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78f80035b0 con 0x7f79101060d0 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.047+0000 7f790e7fc700 1 -- 172.21.15.162:0/2837159592 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3899089756 0 0) 0x7f78f80035b0 con 0x7f79101060d0 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.047+0000 7f790e7fc700 1 -- 172.21.15.162:0/2837159592 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791010b6e0 con 0x7f79101060d0 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.047+0000 7f790e7fc700 1 -- 172.21.15.162:0/2837159592 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f7904001580 con 0x7f79101060d0 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.048+0000 7f790e7fc700 1 -- 172.21.15.162:0/2837159592 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f7904004930 con 0x7f79101060d0 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.048+0000 7f790e7fc700 1 -- 172.21.15.162:0/2837159592 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f7904005960 con 0x7f79101060d0 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.048+0000 7f7915d12700 1 -- 172.21.15.162:0/2837159592 >> v1:172.21.15.162:6789/0 conn(0x7f79101060d0 legacy=0x7f79101064f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.048+0000 7f7915d12700 1 -- 172.21.15.162:0/2837159592 shutdown_connections 2024-04-27T01:05:15.894 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.048+0000 7f7915d12700 1 -- 172.21.15.162:0/2837159592 >> 172.21.15.162:0/2837159592 conn(0x7f7910101630 msgr2=0x7f7910103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.048+0000 7f7915d12700 1 -- 172.21.15.162:0/2837159592 shutdown_connections 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.048+0000 7f7915d12700 1 -- 172.21.15.162:0/2837159592 wait complete. 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.049+0000 7f7915d12700 1 Processor -- start 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.049+0000 7f7915d12700 1 -- start start 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.049+0000 7f7915d12700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f791019fcc0 con 0x7f79101060d0 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.049+0000 7f790f7fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f79101060d0 0x7f791007ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39748/0 (socket says 172.21.15.162:39748) 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.049+0000 7f790f7fe700 1 -- 172.21.15.162:0/2787024011 learned_addr learned my addr 172.21.15.162:0/2787024011 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1924358507 0 0) 0x7f791019fcc0 con 0x7f79101060d0 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78f8005530 con 0x7f79101060d0 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2346652711 0 0) 0x7f78f8005530 con 0x7f79101060d0 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f791019fcc0 con 0x7f79101060d0 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f7904001080 con 0x7f79101060d0 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 872974761 0 0) 0x7f791019fcc0 con 0x7f79101060d0 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79101a0ea0 con 0x7f79101060d0 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f7915d12700 1 -- 172.21.15.162:0/2787024011 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f791019fef0 con 0x7f79101060d0 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f7915d12700 1 -- 172.21.15.162:0/2787024011 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f79101a0430 con 0x7f79101060d0 2024-04-27T01:05:15.895 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f7904004930 con 0x7f79101060d0 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.050+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f7904005fc0 con 0x7f79101060d0 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.051+0000 7f7915d12700 1 -- 172.21.15.162:0/2787024011 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78fc005150 con 0x7f79101060d0 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.051+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 9) ==== 46265+0+0 (unknown 3224811188 0 0) 0x7f7904011630 con 0x7f79101060d0 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.051+0000 7f790effd700 1 -- 172.21.15.162:0/2787024011 >> v1:172.21.15.162:6800/106205076 conn(0x7f78f803a510 legacy=0x7f78f803c9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/106205076 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.051+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7f7904047c10 con 0x7f79101060d0 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.056+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f7904017590 con 0x7f79101060d0 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.252+0000 7f790effd700 1 -- 172.21.15.162:0/2787024011 >> v1:172.21.15.162:6800/106205076 conn(0x7f78f803a510 legacy=0x7f78f803c9d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/106205076 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.322+0000 7f7915d12700 1 -- 172.21.15.162:0/2787024011 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f78fc005c50 con 0x7f79101060d0 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.324+0000 7f790cff9700 1 -- 172.21.15.162:0/2787024011 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7f7904016e90 con 0x7f79101060d0 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.327+0000 7f7915d12700 1 -- 172.21.15.162:0/2787024011 >> v1:172.21.15.162:6800/106205076 conn(0x7f78f803a510 legacy=0x7f78f803c9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.327+0000 7f7915d12700 1 -- 172.21.15.162:0/2787024011 >> v1:172.21.15.162:6789/0 conn(0x7f79101060d0 legacy=0x7f791007ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:15.896 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.327+0000 7f7915d12700 1 -- 172.21.15.162:0/2787024011 shutdown_connections 2024-04-27T01:05:15.897 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.327+0000 7f7915d12700 1 -- 172.21.15.162:0/2787024011 >> 172.21.15.162:0/2787024011 conn(0x7f7910101630 msgr2=0x7f7910103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:15.897 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.332+0000 7f7915d12700 1 -- 172.21.15.162:0/2787024011 shutdown_connections 2024-04-27T01:05:15.897 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:15.332+0000 7f7915d12700 1 -- 172.21.15.162:0/2787024011 wait complete. 2024-04-27T01:05:15.897 INFO:teuthology.orchestra.run.smithi162.stdout:Waiting for the mgr to restart... 2024-04-27T01:05:15.897 INFO:teuthology.orchestra.run.smithi162.stdout:Waiting for mgr epoch 9... 2024-04-27T01:05:16.322 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2787024011' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-27T01:05:16.322 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:16 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:16.150+0000 7ff34da6b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-27T01:05:16.322 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:16 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:16.328+0000 7ff34da6b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-27T01:05:16.592 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:16 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:16.502+0000 7ff34da6b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-27T01:05:17.082 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:16 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:16.791+0000 7ff34da6b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-27T01:05:17.083 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:16 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:16.883+0000 7ff34da6b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-27T01:05:17.083 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:17.088+0000 7ff34da6b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-27T01:05:17.342 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:17.283+0000 7ff34da6b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:05:18.092 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:17.886+0000 7ff34da6b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-27T01:05:18.592 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:18 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:18.313+0000 7ff34da6b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-27T01:05:18.592 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:18 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:18.411+0000 7ff34da6b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-27T01:05:19.092 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:18 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:18.614+0000 7ff34da6b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-27T01:05:19.472 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:19.140+0000 7ff34da6b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-27T01:05:19.473 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:19.233+0000 7ff34da6b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-27T01:05:19.818 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:19.478+0000 7ff34da6b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-27T01:05:19.818 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:19.613+0000 7ff34da6b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-27T01:05:19.819 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:19.707+0000 7ff34da6b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-27T01:05:19.819 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:19.824+0000 7ff34da6b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-27T01:05:20.092 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:19.918+0000 7ff34da6b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-27T01:05:21.002 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:20 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:20.737+0000 7ff34da6b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-27T01:05:21.002 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:05:20 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:05:20.955+0000 7ff34da6b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-27T01:05:21.301 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: Active manager daemon y restarted 2024-04-27T01:05:21.301 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: Activating manager daemon y 2024-04-27T01:05:21.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: osdmap e3: 0 total, 0 up, 0 in 2024-04-27T01:05:21.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: mgrmap e10: y(active, starting, since 0.00327689s) 2024-04-27T01:05:21.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:05:21.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-27T01:05:21.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:05:21.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:05:21.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:05:21.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: Manager daemon y is now available 2024-04-27T01:05:21.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:05:21.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:05:22.178 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:22 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-27T01:05:22.178 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:22 smithi162 ceph-mon[26714]: mgrmap e11: y(active, since 1.00594s) 2024-04-27T01:05:22.886 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout { 2024-04-27T01:05:22.886 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-27T01:05:22.886 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout } 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.355+0000 7fc366761700 1 Processor -- start 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.355+0000 7fc366761700 1 -- start start 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.355+0000 7fc366761700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc36010a520 con 0x7fc3601060f0 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.356+0000 7fc35ffff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc3601060f0 0x7fc360106510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39756/0 (socket says 172.21.15.162:39756) 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.356+0000 7fc35ffff700 1 -- 172.21.15.162:0/18267194 learned_addr learned my addr 172.21.15.162:0/18267194 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.356+0000 7fc35effd700 1 -- 172.21.15.162:0/18267194 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1405160193 0 0) 0x7fc36010a520 con 0x7fc3601060f0 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.356+0000 7fc35effd700 1 -- 172.21.15.162:0/18267194 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc34c0035b0 con 0x7fc3601060f0 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.356+0000 7fc35effd700 1 -- 172.21.15.162:0/18267194 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3102752677 0 0) 0x7fc34c0035b0 con 0x7fc3601060f0 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.356+0000 7fc35effd700 1 -- 172.21.15.162:0/18267194 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36010b700 con 0x7fc3601060f0 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.356+0000 7fc35effd700 1 -- 172.21.15.162:0/18267194 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc348001580 con 0x7fc3601060f0 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.356+0000 7fc35effd700 1 -- 172.21.15.162:0/18267194 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fc348004930 con 0x7fc3601060f0 2024-04-27T01:05:22.887 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.356+0000 7fc35effd700 1 -- 172.21.15.162:0/18267194 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc348005960 con 0x7fc3601060f0 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.357+0000 7fc366761700 1 -- 172.21.15.162:0/18267194 >> v1:172.21.15.162:6789/0 conn(0x7fc3601060f0 legacy=0x7fc360106510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.357+0000 7fc366761700 1 -- 172.21.15.162:0/18267194 shutdown_connections 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.357+0000 7fc366761700 1 -- 172.21.15.162:0/18267194 >> 172.21.15.162:0/18267194 conn(0x7fc360101670 msgr2=0x7fc360103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.357+0000 7fc366761700 1 -- 172.21.15.162:0/18267194 shutdown_connections 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.357+0000 7fc366761700 1 -- 172.21.15.162:0/18267194 wait complete. 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.358+0000 7fc366761700 1 Processor -- start 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.358+0000 7fc366761700 1 -- start start 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.358+0000 7fc366761700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc36019c570 con 0x7fc3601060f0 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.358+0000 7fc35ffff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc3601060f0 0x7fc36019be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39770/0 (socket says 172.21.15.162:39770) 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.358+0000 7fc35ffff700 1 -- 172.21.15.162:0/3166115129 learned_addr learned my addr 172.21.15.162:0/3166115129 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.358+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2751288720 0 0) 0x7fc36019c570 con 0x7fc3601060f0 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.358+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc34c005530 con 0x7fc3601060f0 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.359+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3855011271 0 0) 0x7fc34c005530 con 0x7fc3601060f0 2024-04-27T01:05:22.888 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.359+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc36019c570 con 0x7fc3601060f0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.359+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc348001080 con 0x7fc3601060f0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.359+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 909348784 0 0) 0x7fc36019c570 con 0x7fc3601060f0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.359+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36019c740 con 0x7fc3601060f0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.359+0000 7fc366761700 1 -- 172.21.15.162:0/3166115129 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc36019ca30 con 0x7fc3601060f0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.359+0000 7fc366761700 1 -- 172.21.15.162:0/3166115129 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3601a0580 con 0x7fc3601060f0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.359+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fc348004c10 con 0x7fc3601060f0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.359+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc348006220 con 0x7fc3601060f0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.360+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 9) ==== 46265+0+0 (unknown 3224811188 0 0) 0x7fc3480118b0 con 0x7fc3601060f0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.360+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 --> v1:172.21.15.162:6800/106205076 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc348001080 con 0x7fc34c03a5b0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.360+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 897+0+0 (unknown 810364846 0 0) 0x7fc348048770 con 0x7fc3601060f0 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.360+0000 7fc35f7fe700 1 -- 172.21.15.162:0/3166115129 >> v1:172.21.15.162:6800/106205076 conn(0x7fc34c03a5b0 legacy=0x7fc34c03ca70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/106205076 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.561+0000 7fc35f7fe700 1 -- 172.21.15.162:0/3166115129 >> v1:172.21.15.162:6800/106205076 conn(0x7fc34c03a5b0 legacy=0x7fc34c03ca70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/106205076 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:16.962+0000 7fc35f7fe700 1 -- 172.21.15.162:0/3166115129 >> v1:172.21.15.162:6800/106205076 conn(0x7fc34c03a5b0 legacy=0x7fc34c03ca70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/106205076 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:17.763+0000 7fc35f7fe700 1 -- 172.21.15.162:0/3166115129 >> v1:172.21.15.162:6800/106205076 conn(0x7fc34c03a5b0 legacy=0x7fc34c03ca70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/106205076 2024-04-27T01:05:22.889 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:19.364+0000 7fc35f7fe700 1 -- 172.21.15.162:0/3166115129 >> v1:172.21.15.162:6800/106205076 conn(0x7fc34c03a5b0 legacy=0x7fc34c03ca70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/106205076 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:20.958+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mgrmap(e 10) ==== 46067+0+0 (unknown 3117739899 0 0) 0x7fc3480134a0 con 0x7fc3601060f0 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:20.959+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 >> v1:172.21.15.162:6800/106205076 conn(0x7fc34c03a5b0 legacy=0x7fc34c03ca70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.960+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mgrmap(e 11) ==== 46159+0+0 (unknown 1358981038 0 0) 0x7fc348048df0 con 0x7fc3601060f0 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.960+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 --> v1:172.21.15.162:6800/2166195007 -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc348001080 con 0x7fc34c03ef80 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.961+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (unknown 0 0 4109177037) 0x7fc348001080 con 0x7fc34c03ef80 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.968+0000 7fc366761700 1 -- 172.21.15.162:0/3166115129 --> v1:172.21.15.162:6800/2166195007 -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fc340002600 con 0x7fc34c03ef80 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.968+0000 7fc35d7fa700 1 -- 172.21.15.162:0/3166115129 <== mgr.14152 v1:172.21.15.162:6800/2166195007 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (unknown 0 0 3086460295) 0x7fc340002600 con 0x7fc34c03ef80 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.969+0000 7fc366761700 1 -- 172.21.15.162:0/3166115129 >> v1:172.21.15.162:6800/2166195007 conn(0x7fc34c03ef80 legacy=0x7fc34c041370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.969+0000 7fc366761700 1 -- 172.21.15.162:0/3166115129 >> v1:172.21.15.162:6789/0 conn(0x7fc3601060f0 legacy=0x7fc36019be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.969+0000 7fc366761700 1 -- 172.21.15.162:0/3166115129 shutdown_connections 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.969+0000 7fc366761700 1 -- 172.21.15.162:0/3166115129 >> 172.21.15.162:0/3166115129 conn(0x7fc360101670 msgr2=0x7fc360101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.969+0000 7fc366761700 1 -- 172.21.15.162:0/3166115129 shutdown_connections 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:21.969+0000 7fc366761700 1 -- 172.21.15.162:0/3166115129 wait complete. 2024-04-27T01:05:22.890 INFO:teuthology.orchestra.run.smithi162.stdout:mgr epoch 9 is available 2024-04-27T01:05:22.891 INFO:teuthology.orchestra.run.smithi162.stdout:Generating a dashboard self-signed certificate... 2024-04-27T01:05:23.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:23 smithi162 ceph-mon[26714]: [27/Apr/2024:01:05:21] ENGINE Bus STARTING 2024-04-27T01:05:23.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:23 smithi162 ceph-mon[26714]: [27/Apr/2024:01:05:21] ENGINE Serving on http://172.21.15.162:8765 2024-04-27T01:05:23.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:23 smithi162 ceph-mon[26714]: [27/Apr/2024:01:05:21] ENGINE Serving on https://172.21.15.162:7150 2024-04-27T01:05:23.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:23 smithi162 ceph-mon[26714]: [27/Apr/2024:01:05:21] ENGINE Bus STARTED 2024-04-27T01:05:23.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:23 smithi162 ceph-mon[26714]: from='client.14156 v1:172.21.15.162:0/3166115129' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-27T01:05:23.206 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:23 smithi162 ceph-mon[26714]: from='client.14156 v1:172.21.15.162:0/3166115129' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-27T01:05:23.207 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:24.258 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-27T01:05:24.258 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.315+0000 7fa9d5ebb700 1 Processor -- start 2024-04-27T01:05:24.258 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.315+0000 7fa9d5ebb700 1 -- start start 2024-04-27T01:05:24.258 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.315+0000 7fa9d5ebb700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9d007a340 con 0x7fa9d007b740 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.315+0000 7fa9cf7fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fa9d007b740 0x7fa9d0079ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39828/0 (socket says 172.21.15.162:39828) 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.315+0000 7fa9cf7fe700 1 -- 172.21.15.162:0/3733918918 learned_addr learned my addr 172.21.15.162:0/3733918918 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.315+0000 7fa9ce7fc700 1 -- 172.21.15.162:0/3733918918 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866977622 0 0) 0x7fa9d007a340 con 0x7fa9d007b740 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.315+0000 7fa9ce7fc700 1 -- 172.21.15.162:0/3733918918 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9b80035b0 con 0x7fa9d007b740 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.316+0000 7fa9ce7fc700 1 -- 172.21.15.162:0/3733918918 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3479020427 0 0) 0x7fa9b80035b0 con 0x7fa9d007b740 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.316+0000 7fa9ce7fc700 1 -- 172.21.15.162:0/3733918918 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d007a510 con 0x7fa9d007b740 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.316+0000 7fa9ce7fc700 1 -- 172.21.15.162:0/3733918918 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fa9c0001580 con 0x7fa9d007b740 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.316+0000 7fa9ce7fc700 1 -- 172.21.15.162:0/3733918918 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fa9c0004930 con 0x7fa9d007b740 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.316+0000 7fa9ce7fc700 1 -- 172.21.15.162:0/3733918918 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fa9c0005a70 con 0x7fa9d007b740 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.317+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/3733918918 >> v1:172.21.15.162:6789/0 conn(0x7fa9d007b740 legacy=0x7fa9d0079ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.317+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/3733918918 shutdown_connections 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.317+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/3733918918 >> 172.21.15.162:0/3733918918 conn(0x7fa9d0101670 msgr2=0x7fa9d0103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:24.259 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.317+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/3733918918 shutdown_connections 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.317+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/3733918918 wait complete. 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.317+0000 7fa9d5ebb700 1 Processor -- start 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.318+0000 7fa9d5ebb700 1 -- start start 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.318+0000 7fa9d5ebb700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa9d019c5b0 con 0x7fa9d007b740 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.318+0000 7fa9cf7fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fa9d007b740 0x7fa9d019bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39830/0 (socket says 172.21.15.162:39830) 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.318+0000 7fa9cf7fe700 1 -- 172.21.15.162:0/1710003869 learned_addr learned my addr 172.21.15.162:0/1710003869 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.318+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 893405929 0 0) 0x7fa9d019c5b0 con 0x7fa9d007b740 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.318+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9b8005530 con 0x7fa9d007b740 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.318+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2467466628 0 0) 0x7fa9b8005530 con 0x7fa9d007b740 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.318+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9d019c5b0 con 0x7fa9d007b740 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.318+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fa9c0001080 con 0x7fa9d007b740 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.319+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1588393696 0 0) 0x7fa9d019c5b0 con 0x7fa9d007b740 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.319+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d019c780 con 0x7fa9d007b740 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.319+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/1710003869 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d019ca70 con 0x7fa9d007b740 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.319+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/1710003869 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa9d01a05c0 con 0x7fa9d007b740 2024-04-27T01:05:24.260 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.319+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fa9c0004930 con 0x7fa9d007b740 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.319+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fa9c0005fe0 con 0x7fa9d007b740 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.320+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 11) ==== 46159+0+0 (unknown 1358981038 0 0) 0x7fa9c0011660 con 0x7fa9d007b740 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.320+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/1710003869 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9bc005150 con 0x7fa9d007b740 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.320+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1300520211 0 0) 0x7fa9c00485c0 con 0x7fa9d007b740 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.325+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fa9c00175e0 con 0x7fa9d007b740 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.536+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/1710003869 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fa9bc000bc0 con 0x7fa9b803a3e0 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.567+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mgrmap(e 12) ==== 46265+0+0 (unknown 3173184549 0 0) 0x7fa9c0004c80 con 0x7fa9d007b740 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.668+0000 7fa9ccff9700 1 -- 172.21.15.162:0/1710003869 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (unknown 0 0 3317752739) 0x7fa9bc000bc0 con 0x7fa9b803a3e0 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.670+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/1710003869 >> v1:172.21.15.162:6800/2166195007 conn(0x7fa9b803a3e0 legacy=0x7fa9b803c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.670+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/1710003869 >> v1:172.21.15.162:6789/0 conn(0x7fa9d007b740 legacy=0x7fa9d019bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.670+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/1710003869 shutdown_connections 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.670+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/1710003869 >> 172.21.15.162:0/1710003869 conn(0x7fa9d0101670 msgr2=0x7fa9d010a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.671+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/1710003869 shutdown_connections 2024-04-27T01:05:24.261 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:23.671+0000 7fa9d5ebb700 1 -- 172.21.15.162:0/1710003869 wait complete. 2024-04-27T01:05:24.262 INFO:teuthology.orchestra.run.smithi162.stdout:Creating initial admin user... 2024-04-27T01:05:24.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:24 smithi162 ceph-mon[26714]: from='client.14164 v1:172.21.15.162:0/1710003869' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:24.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:24 smithi162 ceph-mon[26714]: mgrmap e12: y(active, since 2s) 2024-04-27T01:05:24.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:24.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:24.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:25.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:25 smithi162 ceph-mon[26714]: from='client.14166 v1:172.21.15.162:0/3176980985' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:25.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:26.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$xPJ49kZl7dSY7o/VSOeOQ.UD5U/.MLasWZ7NJB0GXQcDRD6ZbcqQy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714179925, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-27T01:05:26.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.661+0000 7fc029778700 1 Processor -- start 2024-04-27T01:05:26.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.661+0000 7fc029778700 1 -- start start 2024-04-27T01:05:26.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.662+0000 7fc029778700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc02407a340 con 0x7fc02407b740 2024-04-27T01:05:26.018 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.662+0000 7fc022ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc02407b740 0x7fc024079ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47886/0 (socket says 172.21.15.162:47886) 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.662+0000 7fc022ffd700 1 -- 172.21.15.162:0/2829023473 learned_addr learned my addr 172.21.15.162:0/2829023473 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.662+0000 7fc021ffb700 1 -- 172.21.15.162:0/2829023473 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4039868938 0 0) 0x7fc02407a340 con 0x7fc02407b740 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.662+0000 7fc021ffb700 1 -- 172.21.15.162:0/2829023473 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc00c0035b0 con 0x7fc02407b740 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.662+0000 7fc021ffb700 1 -- 172.21.15.162:0/2829023473 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2812846504 0 0) 0x7fc00c0035b0 con 0x7fc02407b740 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.662+0000 7fc021ffb700 1 -- 172.21.15.162:0/2829023473 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02407a510 con 0x7fc02407b740 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.662+0000 7fc021ffb700 1 -- 172.21.15.162:0/2829023473 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc014001580 con 0x7fc02407b740 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.662+0000 7fc021ffb700 1 -- 172.21.15.162:0/2829023473 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fc014004930 con 0x7fc02407b740 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.662+0000 7fc021ffb700 1 -- 172.21.15.162:0/2829023473 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc014005a70 con 0x7fc02407b740 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.663+0000 7fc029778700 1 -- 172.21.15.162:0/2829023473 >> v1:172.21.15.162:6789/0 conn(0x7fc02407b740 legacy=0x7fc024079ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.663+0000 7fc029778700 1 -- 172.21.15.162:0/2829023473 shutdown_connections 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.663+0000 7fc029778700 1 -- 172.21.15.162:0/2829023473 >> 172.21.15.162:0/2829023473 conn(0x7fc0241016a0 msgr2=0x7fc024103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.663+0000 7fc029778700 1 -- 172.21.15.162:0/2829023473 shutdown_connections 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.663+0000 7fc029778700 1 -- 172.21.15.162:0/2829023473 wait complete. 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.663+0000 7fc029778700 1 Processor -- start 2024-04-27T01:05:26.019 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc029778700 1 -- start start 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc029778700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc02419c540 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc022ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc02407b740 0x7fc02419be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47898/0 (socket says 172.21.15.162:47898) 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc022ffd700 1 -- 172.21.15.162:0/3176980985 learned_addr learned my addr 172.21.15.162:0/3176980985 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1174961557 0 0) 0x7fc02419c540 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc00c005530 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1747456876 0 0) 0x7fc00c005530 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc02419c540 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc014001080 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3820354528 0 0) 0x7fc02419c540 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02419c710 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc029778700 1 -- 172.21.15.162:0/3176980985 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc02419ca00 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.664+0000 7fc029778700 1 -- 172.21.15.162:0/3176980985 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc0241a0550 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.665+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fc014004930 con 0x7fc02407b740 2024-04-27T01:05:26.020 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.665+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc0140060c0 con 0x7fc02407b740 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.665+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 12) ==== 46265+0+0 (unknown 3173184549 0 0) 0x7fc014011750 con 0x7fc02407b740 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.665+0000 7fc029778700 1 -- 172.21.15.162:0/3176980985 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc010005150 con 0x7fc02407b740 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.665+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1300520211 0 0) 0x7fc014047bb0 con 0x7fc02407b740 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.670+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fc0140176d0 con 0x7fc02407b740 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:24.908+0000 7fc029778700 1 -- 172.21.15.162:0/3176980985 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fc010002430 con 0x7fc00c03a4c0 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:25.131+0000 7fc00bfff700 1 -- 172.21.15.162:0/3176980985 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (unknown 0 0 3030032037) 0x7fc010002430 con 0x7fc00c03a4c0 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:25.133+0000 7fc029778700 1 -- 172.21.15.162:0/3176980985 >> v1:172.21.15.162:6800/2166195007 conn(0x7fc00c03a4c0 legacy=0x7fc00c03c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:25.133+0000 7fc029778700 1 -- 172.21.15.162:0/3176980985 >> v1:172.21.15.162:6789/0 conn(0x7fc02407b740 legacy=0x7fc02419be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:25.133+0000 7fc029778700 1 -- 172.21.15.162:0/3176980985 shutdown_connections 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:25.133+0000 7fc029778700 1 -- 172.21.15.162:0/3176980985 >> 172.21.15.162:0/3176980985 conn(0x7fc0241016a0 msgr2=0x7fc02410a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:25.134+0000 7fc029778700 1 -- 172.21.15.162:0/3176980985 shutdown_connections 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:25.134+0000 7fc029778700 1 -- 172.21.15.162:0/3176980985 wait complete. 2024-04-27T01:05:26.021 INFO:teuthology.orchestra.run.smithi162.stdout:Fetching dashboard port number... 2024-04-27T01:05:26.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/958747976' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-27T01:05:27.520 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stdout 8443 2024-04-27T01:05:27.520 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.504+0000 7fc9d5864700 1 Processor -- start 2024-04-27T01:05:27.520 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.504+0000 7fc9d5864700 1 -- start start 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.504+0000 7fc9d5864700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9d0109410 con 0x7fc9d0104fe0 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.504+0000 7fc9ceffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc9d0104fe0 0x7fc9d0105400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47908/0 (socket says 172.21.15.162:47908) 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.504+0000 7fc9ceffd700 1 -- 172.21.15.162:0/2517891072 learned_addr learned my addr 172.21.15.162:0/2517891072 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.505+0000 7fc9ce7fc700 1 -- 172.21.15.162:0/2517891072 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793019006 0 0) 0x7fc9d0109410 con 0x7fc9d0104fe0 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.505+0000 7fc9ce7fc700 1 -- 172.21.15.162:0/2517891072 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9bc0035b0 con 0x7fc9d0104fe0 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.505+0000 7fc9ce7fc700 1 -- 172.21.15.162:0/2517891072 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4178319261 0 0) 0x7fc9bc0035b0 con 0x7fc9d0104fe0 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.505+0000 7fc9ce7fc700 1 -- 172.21.15.162:0/2517891072 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d010a5f0 con 0x7fc9d0104fe0 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.505+0000 7fc9ce7fc700 1 -- 172.21.15.162:0/2517891072 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc9c0001580 con 0x7fc9d0104fe0 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.505+0000 7fc9ce7fc700 1 -- 172.21.15.162:0/2517891072 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fc9c0004930 con 0x7fc9d0104fe0 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.505+0000 7fc9ce7fc700 1 -- 172.21.15.162:0/2517891072 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc9c0005960 con 0x7fc9d0104fe0 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.506+0000 7fc9d5864700 1 -- 172.21.15.162:0/2517891072 >> v1:172.21.15.162:6789/0 conn(0x7fc9d0104fe0 legacy=0x7fc9d0105400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.506+0000 7fc9d5864700 1 -- 172.21.15.162:0/2517891072 shutdown_connections 2024-04-27T01:05:27.521 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.506+0000 7fc9d5864700 1 -- 172.21.15.162:0/2517891072 >> 172.21.15.162:0/2517891072 conn(0x7fc9d0100560 msgr2=0x7fc9d01029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.506+0000 7fc9d5864700 1 -- 172.21.15.162:0/2517891072 shutdown_connections 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.506+0000 7fc9d5864700 1 -- 172.21.15.162:0/2517891072 wait complete. 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.506+0000 7fc9d5864700 1 Processor -- start 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.506+0000 7fc9d5864700 1 -- start start 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9d5864700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fc9d019a300 con 0x7fc9d0104fe0 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9ceffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc9d0104fe0 0x7fc9d0199bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47912/0 (socket says 172.21.15.162:47912) 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9ceffd700 1 -- 172.21.15.162:0/958747976 learned_addr learned my addr 172.21.15.162:0/958747976 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1596441723 0 0) 0x7fc9d019a300 con 0x7fc9d0104fe0 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9bc005530 con 0x7fc9d0104fe0 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1842353225 0 0) 0x7fc9bc005530 con 0x7fc9d0104fe0 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc9d019a300 con 0x7fc9d0104fe0 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc9c0001080 con 0x7fc9d0104fe0 2024-04-27T01:05:27.522 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 827976837 0 0) 0x7fc9d019a300 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d019a4d0 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9d5864700 1 -- 172.21.15.162:0/958747976 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d019a7c0 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.507+0000 7fc9d5864700 1 -- 172.21.15.162:0/958747976 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9d019e310 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.508+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fc9c0004930 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.508+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fc9c0006060 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.508+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 12) ==== 46265+0+0 (unknown 3173184549 0 0) 0x7fc9c00117b0 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.508+0000 7fc9d5864700 1 -- 172.21.15.162:0/958747976 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b0005120 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.509+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1300520211 0 0) 0x7fc9c0047b30 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.514+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fc9c0017740 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.714+0000 7fc9d5864700 1 -- 172.21.15.162:0/958747976 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fc9b0005410 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.714+0000 7fc9d4862700 1 -- 172.21.15.162:0/958747976 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7fc9c0017040 con 0x7fc9d0104fe0 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.716+0000 7fc9d5864700 1 -- 172.21.15.162:0/958747976 >> v1:172.21.15.162:6800/2166195007 conn(0x7fc9bc03a510 legacy=0x7fc9bc03c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.716+0000 7fc9d5864700 1 -- 172.21.15.162:0/958747976 >> v1:172.21.15.162:6789/0 conn(0x7fc9d0104fe0 legacy=0x7fc9d0199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.717+0000 7fc9d5864700 1 -- 172.21.15.162:0/958747976 shutdown_connections 2024-04-27T01:05:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.717+0000 7fc9d5864700 1 -- 172.21.15.162:0/958747976 >> 172.21.15.162:0/958747976 conn(0x7fc9d0100560 msgr2=0x7fc9d00788e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:27.524 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.717+0000 7fc9d5864700 1 -- 172.21.15.162:0/958747976 shutdown_connections 2024-04-27T01:05:27.524 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:26.717+0000 7fc9d5864700 1 -- 172.21.15.162:0/958747976 wait complete. 2024-04-27T01:05:27.529 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-27T01:05:27.529 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout disabled 2024-04-27T01:05:27.536 INFO:teuthology.orchestra.run.smithi162.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-27T01:05:27.536 INFO:teuthology.orchestra.run.smithi162.stdout:systemctl: stdout inactive 2024-04-27T01:05:27.537 INFO:teuthology.orchestra.run.smithi162.stdout:firewalld.service is not enabled 2024-04-27T01:05:27.537 INFO:teuthology.orchestra.run.smithi162.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-27T01:05:27.539 INFO:teuthology.orchestra.run.smithi162.stdout:Ceph Dashboard is now available at: 2024-04-27T01:05:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: URL: https://smithi162:8443/ 2024-04-27T01:05:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: User: admin 2024-04-27T01:05:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: Password: e3deqxb4pd 2024-04-27T01:05:27.539 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:27.539 INFO:teuthology.orchestra.run.smithi162.stdout:Saving cluster configuration to /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config directory 2024-04-27T01:05:27.539 INFO:teuthology.orchestra.run.smithi162.stdout:Enabling autotune for osd_memory_target 2024-04-27T01:05:28.726 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.950+0000 7fa7f26c5700 1 Processor -- start 2024-04-27T01:05:28.726 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.950+0000 7fa7f26c5700 1 -- start start 2024-04-27T01:05:28.726 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.950+0000 7fa7f26c5700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa7ec07a360 con 0x7fa7ec07b760 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.950+0000 7fa7ebfff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fa7ec07b760 0x7fa7ec079bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47918/0 (socket says 172.21.15.162:47918) 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.950+0000 7fa7ebfff700 1 -- 172.21.15.162:0/1842312295 learned_addr learned my addr 172.21.15.162:0/1842312295 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.950+0000 7fa7eaffd700 1 -- 172.21.15.162:0/1842312295 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4117668296 0 0) 0x7fa7ec07a360 con 0x7fa7ec07b760 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.950+0000 7fa7eaffd700 1 -- 172.21.15.162:0/1842312295 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d40035b0 con 0x7fa7ec07b760 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.951+0000 7fa7eaffd700 1 -- 172.21.15.162:0/1842312295 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4264163531 0 0) 0x7fa7d40035b0 con 0x7fa7ec07b760 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.951+0000 7fa7eaffd700 1 -- 172.21.15.162:0/1842312295 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec07a530 con 0x7fa7ec07b760 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.951+0000 7fa7eaffd700 1 -- 172.21.15.162:0/1842312295 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fa7dc001580 con 0x7fa7ec07b760 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.951+0000 7fa7eaffd700 1 -- 172.21.15.162:0/1842312295 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fa7dc004930 con 0x7fa7ec07b760 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.951+0000 7fa7eaffd700 1 -- 172.21.15.162:0/1842312295 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fa7dc005a70 con 0x7fa7ec07b760 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.951+0000 7fa7f26c5700 1 -- 172.21.15.162:0/1842312295 >> v1:172.21.15.162:6789/0 conn(0x7fa7ec07b760 legacy=0x7fa7ec079bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.951+0000 7fa7f26c5700 1 -- 172.21.15.162:0/1842312295 shutdown_connections 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.951+0000 7fa7f26c5700 1 -- 172.21.15.162:0/1842312295 >> 172.21.15.162:0/1842312295 conn(0x7fa7ec1016a0 msgr2=0x7fa7ec103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.951+0000 7fa7f26c5700 1 -- 172.21.15.162:0/1842312295 shutdown_connections 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.951+0000 7fa7f26c5700 1 -- 172.21.15.162:0/1842312295 wait complete. 2024-04-27T01:05:28.727 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.952+0000 7fa7f26c5700 1 Processor -- start 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.952+0000 7fa7f26c5700 1 -- start start 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.952+0000 7fa7f26c5700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa7ec19c530 con 0x7fa7ec07b760 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.952+0000 7fa7ebfff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fa7ec07b760 0x7fa7ec19be20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47922/0 (socket says 172.21.15.162:47922) 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.952+0000 7fa7ebfff700 1 -- 172.21.15.162:0/4288919400 learned_addr learned my addr 172.21.15.162:0/4288919400 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2252855361 0 0) 0x7fa7ec19c530 con 0x7fa7ec07b760 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7d4005530 con 0x7fa7ec07b760 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3927040441 0 0) 0x7fa7d4005530 con 0x7fa7ec07b760 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7ec19c530 con 0x7fa7ec07b760 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fa7dc001080 con 0x7fa7ec07b760 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1063800367 0 0) 0x7fa7ec19c530 con 0x7fa7ec07b760 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec19c700 con 0x7fa7ec07b760 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7f26c5700 1 -- 172.21.15.162:0/4288919400 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7ec19c9f0 con 0x7fa7ec07b760 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7f26c5700 1 -- 172.21.15.162:0/4288919400 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7ec1a0540 con 0x7fa7ec07b760 2024-04-27T01:05:28.728 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fa7dc004930 con 0x7fa7ec07b760 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.953+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fa7dc005fe0 con 0x7fa7ec07b760 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.954+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7fa7dc011770 con 0x7fa7ec07b760 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.954+0000 7fa7f26c5700 1 -- 172.21.15.162:0/4288919400 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7d8005150 con 0x7fa7ec07b760 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.954+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1300520211 0 0) 0x7fa7dc047b00 con 0x7fa7ec07b760 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:27.959+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fa7dc0177b0 con 0x7fa7ec07b760 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:28.155+0000 7fa7f26c5700 1 -- 172.21.15.162:0/4288919400 --> v1:172.21.15.162:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fa7d8005440 con 0x7fa7ec07b760 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:28.156+0000 7fa7e97fa700 1 -- 172.21.15.162:0/4288919400 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (unknown 689008156 0 0) 0x7fa7dc0170b0 con 0x7fa7ec07b760 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:28.158+0000 7fa7f26c5700 1 -- 172.21.15.162:0/4288919400 >> v1:172.21.15.162:6800/2166195007 conn(0x7fa7d403a560 legacy=0x7fa7d403ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:28.158+0000 7fa7f26c5700 1 -- 172.21.15.162:0/4288919400 >> v1:172.21.15.162:6789/0 conn(0x7fa7ec07b760 legacy=0x7fa7ec19be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:28.158+0000 7fa7f26c5700 1 -- 172.21.15.162:0/4288919400 shutdown_connections 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:28.158+0000 7fa7f26c5700 1 -- 172.21.15.162:0/4288919400 >> 172.21.15.162:0/4288919400 conn(0x7fa7ec1016a0 msgr2=0x7fa7ec10a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:28.158+0000 7fa7f26c5700 1 -- 172.21.15.162:0/4288919400 shutdown_connections 2024-04-27T01:05:28.729 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:28.158+0000 7fa7f26c5700 1 -- 172.21.15.162:0/4288919400 wait complete. 2024-04-27T01:05:28.974 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:28 smithi162 ceph-mon[26714]: mgrmap e13: y(active, since 6s) 2024-04-27T01:05:30.057 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.174+0000 7f3a5678f700 1 Processor -- start 2024-04-27T01:05:30.057 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.174+0000 7f3a5678f700 1 -- start start 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.174+0000 7f3a5678f700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3a5007a360 con 0x7f3a5007b760 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.174+0000 7f3a4ffff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f3a5007b760 0x7f3a50079bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47936/0 (socket says 172.21.15.162:47936) 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.174+0000 7f3a4ffff700 1 -- 172.21.15.162:0/3113978012 learned_addr learned my addr 172.21.15.162:0/3113978012 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.174+0000 7f3a4effd700 1 -- 172.21.15.162:0/3113978012 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 758323370 0 0) 0x7f3a5007a360 con 0x7f3a5007b760 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.174+0000 7f3a4effd700 1 -- 172.21.15.162:0/3113978012 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a380035b0 con 0x7f3a5007b760 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.175+0000 7f3a4effd700 1 -- 172.21.15.162:0/3113978012 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 435698924 0 0) 0x7f3a380035b0 con 0x7f3a5007b760 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.175+0000 7f3a4effd700 1 -- 172.21.15.162:0/3113978012 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5007a530 con 0x7f3a5007b760 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.175+0000 7f3a4effd700 1 -- 172.21.15.162:0/3113978012 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f3a40001580 con 0x7f3a5007b760 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.175+0000 7f3a4effd700 1 -- 172.21.15.162:0/3113978012 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f3a40004930 con 0x7f3a5007b760 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.175+0000 7f3a4effd700 1 -- 172.21.15.162:0/3113978012 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f3a40005a70 con 0x7f3a5007b760 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.175+0000 7f3a5678f700 1 -- 172.21.15.162:0/3113978012 >> v1:172.21.15.162:6789/0 conn(0x7f3a5007b760 legacy=0x7f3a50079bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.175+0000 7f3a5678f700 1 -- 172.21.15.162:0/3113978012 shutdown_connections 2024-04-27T01:05:30.058 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.175+0000 7f3a5678f700 1 -- 172.21.15.162:0/3113978012 >> 172.21.15.162:0/3113978012 conn(0x7f3a50101630 msgr2=0x7f3a50103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.175+0000 7f3a5678f700 1 -- 172.21.15.162:0/3113978012 shutdown_connections 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.175+0000 7f3a5678f700 1 -- 172.21.15.162:0/3113978012 wait complete. 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.176+0000 7f3a5678f700 1 Processor -- start 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.176+0000 7f3a5678f700 1 -- start start 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.176+0000 7f3a5678f700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3a5019c5f0 con 0x7f3a5007b760 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.176+0000 7f3a4ffff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f3a5007b760 0x7f3a5019bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47944/0 (socket says 172.21.15.162:47944) 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.176+0000 7f3a4ffff700 1 -- 172.21.15.162:0/1047556191 learned_addr learned my addr 172.21.15.162:0/1047556191 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2657308002 0 0) 0x7f3a5019c5f0 con 0x7f3a5007b760 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a38005530 con 0x7f3a5007b760 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2060505788 0 0) 0x7f3a38005530 con 0x7f3a5007b760 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3a5019c5f0 con 0x7f3a5007b760 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f3a40001080 con 0x7f3a5007b760 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 596932117 0 0) 0x7f3a5019c5f0 con 0x7f3a5007b760 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5019c7c0 con 0x7f3a5007b760 2024-04-27T01:05:30.059 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a5678f700 1 -- 172.21.15.162:0/1047556191 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5019cab0 con 0x7f3a5007b760 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a5678f700 1 -- 172.21.15.162:0/1047556191 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3a501a0600 con 0x7f3a5007b760 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f3a40004930 con 0x7f3a5007b760 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.177+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f3a40005fe0 con 0x7f3a5007b760 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.178+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7f3a400116f0 con 0x7f3a5007b760 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.178+0000 7f3a5678f700 1 -- 172.21.15.162:0/1047556191 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3c005150 con 0x7f3a5007b760 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.178+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1300520211 0 0) 0x7f3a40047ae0 con 0x7f3a5007b760 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.183+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f3a40017710 con 0x7f3a5007b760 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.490+0000 7f3a5678f700 1 -- 172.21.15.162:0/1047556191 --> v1:172.21.15.162:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f3a3c005440 con 0x7f3a5007b760 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.492+0000 7f3a4d7fa700 1 -- 172.21.15.162:0/1047556191 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (unknown 1169358022 0 0) 0x7f3a40017010 con 0x7f3a5007b760 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.494+0000 7f3a5678f700 1 -- 172.21.15.162:0/1047556191 >> v1:172.21.15.162:6800/2166195007 conn(0x7f3a3803a5a0 legacy=0x7f3a3803ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.494+0000 7f3a5678f700 1 -- 172.21.15.162:0/1047556191 >> v1:172.21.15.162:6789/0 conn(0x7f3a5007b760 legacy=0x7f3a5019bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.495+0000 7f3a5678f700 1 -- 172.21.15.162:0/1047556191 shutdown_connections 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.495+0000 7f3a5678f700 1 -- 172.21.15.162:0/1047556191 >> 172.21.15.162:0/1047556191 conn(0x7f3a50101630 msgr2=0x7f3a5010af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.495+0000 7f3a5678f700 1 -- 172.21.15.162:0/1047556191 shutdown_connections 2024-04-27T01:05:30.060 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr 2024-04-27T01:05:29.495+0000 7f3a5678f700 1 -- 172.21.15.162:0/1047556191 wait complete. 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout: sudo /sbin/cephadm shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout:Or, if you are only running a single cluster on this host: 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout: sudo /sbin/cephadm shell 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout: ceph telemetry on 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout:For more information see: 2024-04-27T01:05:30.061 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:30.062 INFO:teuthology.orchestra.run.smithi162.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-27T01:05:30.062 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:30.062 INFO:teuthology.orchestra.run.smithi162.stdout:Bootstrap complete. 2024-04-27T01:05:30.097 INFO:tasks.cephadm:Fetching config... 2024-04-27T01:05:30.097 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:05:30.097 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-27T01:05:30.113 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-27T01:05:30.113 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:05:30.113 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-27T01:05:30.169 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-27T01:05:30.169 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:05:30.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-27T01:05:30.232 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-27T01:05:30.232 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:05:30.233 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-27T01:05:30.287 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-27T01:05:30.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllRkhIxUhtQJPpMVc/E/h5zKwOw+MQ6sel0XMkgLcobJp/5fLUSPzqSwnByyfPjMal88vHv9UKkssBdQBdJiMAhuQ+yn3Y4+2pe7B2lLjGjnnfbfCw20OJuNffy3H2i6gp3fZ5uhKH0oQnrnQ/3iBvr0797LzjYNfqc/SwYQbgUmehnA8vAAuTv2aUQEpYbVixCTgRbudUUs565ZnlMr74+ZWGvDlYyEW7PsR5A0k1VCAz7fwvMW0sCt1GMUSArbh6Di9M1Xt6LIbxbnf7Vrq0OrCeLeQmzo+NrKmr5C6iKciVnv8zRr905ASjUjKSjmrW+tXTD/2DnnaF10NmAk5BXkFqa38PRBZnQGOHCFzNRvK8QyupSiGDC4Qa05ywZTQ5/1mDjl20vcK4kuQFSdhpar58FpO8QoVL6XyuHCcfYt0GCWWK5w44nk95sNgOuIzBuxy3h97SMnR6fFf9cJ/oQeeQL2Rny5hgtGYFAK8Sza5iHYW+3chEO8qhyJQfc0= ceph-ea378f3c-0431-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-27T01:05:30.359 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllRkhIxUhtQJPpMVc/E/h5zKwOw+MQ6sel0XMkgLcobJp/5fLUSPzqSwnByyfPjMal88vHv9UKkssBdQBdJiMAhuQ+yn3Y4+2pe7B2lLjGjnnfbfCw20OJuNffy3H2i6gp3fZ5uhKH0oQnrnQ/3iBvr0797LzjYNfqc/SwYQbgUmehnA8vAAuTv2aUQEpYbVixCTgRbudUUs565ZnlMr74+ZWGvDlYyEW7PsR5A0k1VCAz7fwvMW0sCt1GMUSArbh6Di9M1Xt6LIbxbnf7Vrq0OrCeLeQmzo+NrKmr5C6iKciVnv8zRr905ASjUjKSjmrW+tXTD/2DnnaF10NmAk5BXkFqa38PRBZnQGOHCFzNRvK8QyupSiGDC4Qa05ywZTQ5/1mDjl20vcK4kuQFSdhpar58FpO8QoVL6XyuHCcfYt0GCWWK5w44nk95sNgOuIzBuxy3h97SMnR6fFf9cJ/oQeeQL2Rny5hgtGYFAK8Sza5iHYW+3chEO8qhyJQfc0= ceph-ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:05:30.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllRkhIxUhtQJPpMVc/E/h5zKwOw+MQ6sel0XMkgLcobJp/5fLUSPzqSwnByyfPjMal88vHv9UKkssBdQBdJiMAhuQ+yn3Y4+2pe7B2lLjGjnnfbfCw20OJuNffy3H2i6gp3fZ5uhKH0oQnrnQ/3iBvr0797LzjYNfqc/SwYQbgUmehnA8vAAuTv2aUQEpYbVixCTgRbudUUs565ZnlMr74+ZWGvDlYyEW7PsR5A0k1VCAz7fwvMW0sCt1GMUSArbh6Di9M1Xt6LIbxbnf7Vrq0OrCeLeQmzo+NrKmr5C6iKciVnv8zRr905ASjUjKSjmrW+tXTD/2DnnaF10NmAk5BXkFqa38PRBZnQGOHCFzNRvK8QyupSiGDC4Qa05ywZTQ5/1mDjl20vcK4kuQFSdhpar58FpO8QoVL6XyuHCcfYt0GCWWK5w44nk95sNgOuIzBuxy3h97SMnR6fFf9cJ/oQeeQL2Rny5hgtGYFAK8Sza5iHYW+3chEO8qhyJQfc0= ceph-ea378f3c-0431-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-27T01:05:30.408 INFO:teuthology.orchestra.run.smithi181.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllRkhIxUhtQJPpMVc/E/h5zKwOw+MQ6sel0XMkgLcobJp/5fLUSPzqSwnByyfPjMal88vHv9UKkssBdQBdJiMAhuQ+yn3Y4+2pe7B2lLjGjnnfbfCw20OJuNffy3H2i6gp3fZ5uhKH0oQnrnQ/3iBvr0797LzjYNfqc/SwYQbgUmehnA8vAAuTv2aUQEpYbVixCTgRbudUUs565ZnlMr74+ZWGvDlYyEW7PsR5A0k1VCAz7fwvMW0sCt1GMUSArbh6Di9M1Xt6LIbxbnf7Vrq0OrCeLeQmzo+NrKmr5C6iKciVnv8zRr905ASjUjKSjmrW+tXTD/2DnnaF10NmAk5BXkFqa38PRBZnQGOHCFzNRvK8QyupSiGDC4Qa05ywZTQ5/1mDjl20vcK4kuQFSdhpar58FpO8QoVL6XyuHCcfYt0GCWWK5w44nk95sNgOuIzBuxy3h97SMnR6fFf9cJ/oQeeQL2Rny5hgtGYFAK8Sza5iHYW+3chEO8qhyJQfc0= ceph-ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:05:30.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-27T01:05:30.682 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:05:30.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1047556191' entity='client.admin' 2024-04-27T01:05:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.139+0000 7f613d2dc700 1 -- 172.21.15.162:0/2975101770 >> v1:172.21.15.162:6789/0 conn(0x7f6138100b00 legacy=0x7f6138100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.139+0000 7f613d2dc700 1 -- 172.21.15.162:0/2975101770 shutdown_connections 2024-04-27T01:05:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.139+0000 7f613d2dc700 1 -- 172.21.15.162:0/2975101770 >> 172.21.15.162:0/2975101770 conn(0x7f61380fc0a0 msgr2=0x7f61380fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.139+0000 7f613d2dc700 1 -- 172.21.15.162:0/2975101770 shutdown_connections 2024-04-27T01:05:33.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.139+0000 7f613d2dc700 1 -- 172.21.15.162:0/2975101770 wait complete. 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.140+0000 7f613d2dc700 1 Processor -- start 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.140+0000 7f613d2dc700 1 -- start start 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.140+0000 7f613d2dc700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6138195be0 con 0x7f6138100b00 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.140+0000 7f6136ffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6138100b00 0x7f61381954d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47958/0 (socket says 172.21.15.162:47958) 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.140+0000 7f6136ffd700 1 -- 172.21.15.162:0/2469141932 learned_addr learned my addr 172.21.15.162:0/2469141932 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1471469896 0 0) 0x7f6138195be0 con 0x7f6138100b00 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6120005530 con 0x7f6138100b00 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3629424064 0 0) 0x7f6120005530 con 0x7f6138100b00 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6138195be0 con 0x7f6138100b00 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f6128001480 con 0x7f6138100b00 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 356924286 0 0) 0x7f6138195be0 con 0x7f6138100b00 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6138195db0 con 0x7f6138100b00 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f613d2dc700 1 -- 172.21.15.162:0/2469141932 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6138196040 con 0x7f6138100b00 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f613d2dc700 1 -- 172.21.15.162:0/2469141932 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f613806a240 con 0x7f6138100b00 2024-04-27T01:05:33.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f6128001b60 con 0x7f6138100b00 2024-04-27T01:05:33.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.141+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f6128004d20 con 0x7f6138100b00 2024-04-27T01:05:33.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.142+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7f6128004fa0 con 0x7f6138100b00 2024-04-27T01:05:33.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.142+0000 7f613d2dc700 1 -- 172.21.15.162:0/2469141932 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6124005150 con 0x7f6138100b00 2024-04-27T01:05:33.137 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:33.137 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:33.137 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:05:33.137 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:33.137 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:05:33.137 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:33.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.142+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1300520211 0 0) 0x7f61280479c0 con 0x7f6138100b00 2024-04-27T01:05:33.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.147+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f6128017590 con 0x7f6138100b00 2024-04-27T01:05:33.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.347+0000 7f613d2dc700 1 -- 172.21.15.162:0/2469141932 --> v1:172.21.15.162:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f6124005440 con 0x7f6138100b00 2024-04-27T01:05:33.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.349+0000 7f611ffff700 1 -- 172.21.15.162:0/2469141932 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (unknown 3028693289 0 0) 0x7f6128016e90 con 0x7f6138100b00 2024-04-27T01:05:33.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.351+0000 7f613d2dc700 1 -- 172.21.15.162:0/2469141932 >> v1:172.21.15.162:6800/2166195007 conn(0x7f612003c630 legacy=0x7f612003eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:33.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.351+0000 7f613d2dc700 1 -- 172.21.15.162:0/2469141932 >> v1:172.21.15.162:6789/0 conn(0x7f6138100b00 legacy=0x7f61381954d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:33.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.351+0000 7f613d2dc700 1 -- 172.21.15.162:0/2469141932 shutdown_connections 2024-04-27T01:05:33.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.352+0000 7f613d2dc700 1 -- 172.21.15.162:0/2469141932 >> 172.21.15.162:0/2469141932 conn(0x7f61380fc0a0 msgr2=0x7f61380fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:33.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.352+0000 7f613d2dc700 1 -- 172.21.15.162:0/2469141932 shutdown_connections 2024-04-27T01:05:33.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:33.352+0000 7f613d2dc700 1 -- 172.21.15.162:0/2469141932 wait complete. 2024-04-27T01:05:33.974 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-27T01:05:33.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-27T01:05:34.505 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2469141932' entity='client.admin' 2024-04-27T01:05:34.505 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:34 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:05:34.505 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:34 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:34.638 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:05:36.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.111+0000 7f4e5e208700 1 -- 172.21.15.162:0/2944991534 >> v1:172.21.15.162:6789/0 conn(0x7f4e581031d0 legacy=0x7f4e581035f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:36.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.111+0000 7f4e5e208700 1 -- 172.21.15.162:0/2944991534 shutdown_connections 2024-04-27T01:05:36.107 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.111+0000 7f4e5e208700 1 -- 172.21.15.162:0/2944991534 >> 172.21.15.162:0/2944991534 conn(0x7f4e580fb810 msgr2=0x7f4e580fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.111+0000 7f4e5e208700 1 -- 172.21.15.162:0/2944991534 shutdown_connections 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.111+0000 7f4e5e208700 1 -- 172.21.15.162:0/2944991534 wait complete. 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.112+0000 7f4e5e208700 1 Processor -- start 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.112+0000 7f4e5e208700 1 -- start start 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.112+0000 7f4e5e208700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4e58196d20 con 0x7f4e581031d0 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.112+0000 7f4e577fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f4e581031d0 0x7f4e58196610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:36998/0 (socket says 172.21.15.162:36998) 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.112+0000 7f4e577fe700 1 -- 172.21.15.162:0/3528099135 learned_addr learned my addr 172.21.15.162:0/3528099135 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145926780 0 0) 0x7f4e58196d20 con 0x7f4e581031d0 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e40005530 con 0x7f4e581031d0 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2438926454 0 0) 0x7f4e40005530 con 0x7f4e581031d0 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4e58196d20 con 0x7f4e581031d0 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f4e48004750 con 0x7f4e581031d0 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2851408801 0 0) 0x7f4e58196d20 con 0x7f4e581031d0 2024-04-27T01:05:36.108 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e58196ef0 con 0x7f4e581031d0 2024-04-27T01:05:36.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e5e208700 1 -- 172.21.15.162:0/3528099135 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4e581971e0 con 0x7f4e581031d0 2024-04-27T01:05:36.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e5e208700 1 -- 172.21.15.162:0/3528099135 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4e5819ad30 con 0x7f4e581031d0 2024-04-27T01:05:36.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f4e48001190 con 0x7f4e581031d0 2024-04-27T01:05:36.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.113+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f4e48004cb0 con 0x7f4e581031d0 2024-04-27T01:05:36.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.114+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7f4e48004e70 con 0x7f4e581031d0 2024-04-27T01:05:36.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.114+0000 7f4e5e208700 1 -- 172.21.15.162:0/3528099135 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e44005150 con 0x7f4e581031d0 2024-04-27T01:05:36.109 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.114+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1300520211 0 0) 0x7f4e48047910 con 0x7f4e581031d0 2024-04-27T01:05:36.111 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.119+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f4e48017500 con 0x7f4e581031d0 2024-04-27T01:05:36.311 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.318+0000 7f4e5e208700 1 -- 172.21.15.162:0/3528099135 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f4e44000bc0 con 0x7f4e4003a420 2024-04-27T01:05:36.314 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.321+0000 7f4e54ff9700 1 -- 172.21.15.162:0/3528099135 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (unknown 0 0 0) 0x7f4e44000bc0 con 0x7f4e4003a420 2024-04-27T01:05:36.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.323+0000 7f4e5e208700 1 -- 172.21.15.162:0/3528099135 >> v1:172.21.15.162:6800/2166195007 conn(0x7f4e4003a420 legacy=0x7f4e4003c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:36.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.323+0000 7f4e5e208700 1 -- 172.21.15.162:0/3528099135 >> v1:172.21.15.162:6789/0 conn(0x7f4e581031d0 legacy=0x7f4e58196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:36.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.323+0000 7f4e5e208700 1 -- 172.21.15.162:0/3528099135 shutdown_connections 2024-04-27T01:05:36.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.323+0000 7f4e5e208700 1 -- 172.21.15.162:0/3528099135 >> 172.21.15.162:0/3528099135 conn(0x7f4e580fb810 msgr2=0x7f4e580fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:36.316 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.323+0000 7f4e5e208700 1 -- 172.21.15.162:0/3528099135 shutdown_connections 2024-04-27T01:05:36.317 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:36.323+0000 7f4e5e208700 1 -- 172.21.15.162:0/3528099135 wait complete. 2024-04-27T01:05:37.059 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi181 2024-04-27T01:05:37.060 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:05:37.060 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.conf 2024-04-27T01:05:37.078 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:05:37.078 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:05:37.135 INFO:tasks.cephadm:Adding host smithi181 to orchestrator... 2024-04-27T01:05:37.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch host add smithi181 2024-04-27T01:05:37.380 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:05:37.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: from='client.14176 v1:172.21.15.162:0/3528099135' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:37.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:37.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:05:37.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:05:37.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:05:37.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: Updating smithi162:/etc/ceph/ceph.conf 2024-04-27T01:05:37.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:05:37.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:05:37.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:05:37.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:37.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:37.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:38.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.830+0000 7f693a77e700 1 -- 172.21.15.162:0/3255380793 >> v1:172.21.15.162:6789/0 conn(0x7f6934101dc0 legacy=0x7f69341021a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:38.823 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.830+0000 7f693a77e700 1 -- 172.21.15.162:0/3255380793 shutdown_connections 2024-04-27T01:05:38.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.830+0000 7f693a77e700 1 -- 172.21.15.162:0/3255380793 >> 172.21.15.162:0/3255380793 conn(0x7f69340fd690 msgr2=0x7f69340ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:38.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.830+0000 7f693a77e700 1 -- 172.21.15.162:0/3255380793 shutdown_connections 2024-04-27T01:05:38.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.830+0000 7f693a77e700 1 -- 172.21.15.162:0/3255380793 wait complete. 2024-04-27T01:05:38.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.831+0000 7f693a77e700 1 Processor -- start 2024-04-27T01:05:38.824 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.831+0000 7f693a77e700 1 -- start start 2024-04-27T01:05:38.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.831+0000 7f693a77e700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f693419cd70 con 0x7f6934101dc0 2024-04-27T01:05:38.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.831+0000 7f6933fff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6934101dc0 0x7f693419c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:37004/0 (socket says 172.21.15.162:37004) 2024-04-27T01:05:38.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.831+0000 7f6933fff700 1 -- 172.21.15.162:0/4117682193 learned_addr learned my addr 172.21.15.162:0/4117682193 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:38.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.831+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 60658645 0 0) 0x7f693419cd70 con 0x7f6934101dc0 2024-04-27T01:05:38.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.831+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f691c005530 con 0x7f6934101dc0 2024-04-27T01:05:38.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.831+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3071277808 0 0) 0x7f691c005530 con 0x7f6934101dc0 2024-04-27T01:05:38.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.831+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f693419cd70 con 0x7f6934101dc0 2024-04-27T01:05:38.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.831+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f6924004700 con 0x7f6934101dc0 2024-04-27T01:05:38.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.832+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3910899497 0 0) 0x7f693419cd70 con 0x7f6934101dc0 2024-04-27T01:05:38.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.832+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693419cf40 con 0x7f6934101dc0 2024-04-27T01:05:38.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.832+0000 7f693a77e700 1 -- 172.21.15.162:0/4117682193 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f693419f3f0 con 0x7f6934101dc0 2024-04-27T01:05:38.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.832+0000 7f693a77e700 1 -- 172.21.15.162:0/4117682193 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f693419f790 con 0x7f6934101dc0 2024-04-27T01:05:38.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.832+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f6924003e60 con 0x7f6934101dc0 2024-04-27T01:05:38.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.832+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f6924003760 con 0x7f6934101dc0 2024-04-27T01:05:38.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.832+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7f6924005020 con 0x7f6934101dc0 2024-04-27T01:05:38.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.833+0000 7f693a77e700 1 -- 172.21.15.162:0/4117682193 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6920005120 con 0x7f6934101dc0 2024-04-27T01:05:38.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.833+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1300520211 0 0) 0x7f6924047a20 con 0x7f6934101dc0 2024-04-27T01:05:38.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:38.837+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f6924017590 con 0x7f6934101dc0 2024-04-27T01:05:39.029 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:39.036+0000 7f693a77e700 1 -- 172.21.15.162:0/4117682193 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}) -- 0x7f6920000b90 con 0x7f691c03a570 2024-04-27T01:05:39.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:39 smithi162 ceph-mon[26714]: from='client.14178 v1:172.21.15.162:0/4117682193' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:40.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:40 smithi162 ceph-mon[26714]: Deploying cephadm binary to smithi181 2024-04-27T01:05:42.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:41 smithi162 ceph-mon[26714]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:43.127 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:43.134+0000 7f69317fa700 1 -- 172.21.15.162:0/4117682193 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (unknown 0 0 2353736366) 0x7f6920000b90 con 0x7f691c03a570 2024-04-27T01:05:43.128 INFO:teuthology.orchestra.run.smithi162.stdout:Added host 'smithi181' with addr '172.21.15.181' 2024-04-27T01:05:43.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:43.137+0000 7f693a77e700 1 -- 172.21.15.162:0/4117682193 >> v1:172.21.15.162:6800/2166195007 conn(0x7f691c03a570 legacy=0x7f691c03ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:43.130 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:43.137+0000 7f693a77e700 1 -- 172.21.15.162:0/4117682193 >> v1:172.21.15.162:6789/0 conn(0x7f6934101dc0 legacy=0x7f693419c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:43.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:43.137+0000 7f693a77e700 1 -- 172.21.15.162:0/4117682193 shutdown_connections 2024-04-27T01:05:43.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:43.137+0000 7f693a77e700 1 -- 172.21.15.162:0/4117682193 >> 172.21.15.162:0/4117682193 conn(0x7f69340fd690 msgr2=0x7f6934108c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:43.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:43.138+0000 7f693a77e700 1 -- 172.21.15.162:0/4117682193 shutdown_connections 2024-04-27T01:05:43.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:43.138+0000 7f693a77e700 1 -- 172.21.15.162:0/4117682193 wait complete. 2024-04-27T01:05:43.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-27T01:05:44.035 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:05:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:43 smithi162 ceph-mon[26714]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:43 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:43 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:05:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:43 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:45.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:44 smithi162 ceph-mon[26714]: Added host smithi181 2024-04-27T01:05:45.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:44 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.608+0000 7f5fc18ed700 1 -- 172.21.15.162:0/3135145587 >> v1:172.21.15.162:6789/0 conn(0x7f5fbc101db0 legacy=0x7f5fbc102190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.608+0000 7f5fc18ed700 1 -- 172.21.15.162:0/3135145587 shutdown_connections 2024-04-27T01:05:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.608+0000 7f5fc18ed700 1 -- 172.21.15.162:0/3135145587 >> 172.21.15.162:0/3135145587 conn(0x7f5fbc0fd660 msgr2=0x7f5fbc0ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.608+0000 7f5fc18ed700 1 -- 172.21.15.162:0/3135145587 shutdown_connections 2024-04-27T01:05:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.608+0000 7f5fc18ed700 1 -- 172.21.15.162:0/3135145587 wait complete. 2024-04-27T01:05:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.609+0000 7f5fc18ed700 1 Processor -- start 2024-04-27T01:05:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.609+0000 7f5fc18ed700 1 -- start start 2024-04-27T01:05:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.609+0000 7f5fc18ed700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5fbc19cd30 con 0x7f5fbc101db0 2024-04-27T01:05:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.609+0000 7f5fbaffd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f5fbc101db0 0x7f5fbc19c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:50370/0 (socket says 172.21.15.162:50370) 2024-04-27T01:05:45.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.609+0000 7f5fbaffd700 1 -- 172.21.15.162:0/2419474502 learned_addr learned my addr 172.21.15.162:0/2419474502 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.609+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 840346710 0 0) 0x7f5fbc19cd30 con 0x7f5fbc101db0 2024-04-27T01:05:45.603 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.609+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fa4005530 con 0x7f5fbc101db0 2024-04-27T01:05:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.609+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2194347828 0 0) 0x7f5fa4005530 con 0x7f5fbc101db0 2024-04-27T01:05:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.609+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fbc19cd30 con 0x7f5fbc101db0 2024-04-27T01:05:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.609+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f5fb0004700 con 0x7f5fbc101db0 2024-04-27T01:05:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.610+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3699511881 0 0) 0x7f5fbc19cd30 con 0x7f5fbc101db0 2024-04-27T01:05:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.610+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fbc19cf00 con 0x7f5fbc101db0 2024-04-27T01:05:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.610+0000 7f5fc18ed700 1 -- 172.21.15.162:0/2419474502 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5fbc19f3b0 con 0x7f5fbc101db0 2024-04-27T01:05:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.610+0000 7f5fc18ed700 1 -- 172.21.15.162:0/2419474502 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5fbc19f750 con 0x7f5fbc101db0 2024-04-27T01:05:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.610+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f5fb0003760 con 0x7f5fbc101db0 2024-04-27T01:05:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.610+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f5fb0004d40 con 0x7f5fbc101db0 2024-04-27T01:05:45.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.611+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7f5fb0004f00 con 0x7f5fbc101db0 2024-04-27T01:05:45.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.611+0000 7f5fc18ed700 1 -- 172.21.15.162:0/2419474502 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa8005120 con 0x7f5fbc101db0 2024-04-27T01:05:45.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.611+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1300520211 0 0) 0x7f5fb00479c0 con 0x7f5fbc101db0 2024-04-27T01:05:45.608 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.616+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f5fb00174a0 con 0x7f5fbc101db0 2024-04-27T01:05:45.810 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.818+0000 7f5fc18ed700 1 -- 172.21.15.162:0/2419474502 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5fa8000b90 con 0x7f5fa403a420 2024-04-27T01:05:45.812 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.819+0000 7f5fc08eb700 1 -- 172.21.15.162:0/2419474502 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (unknown 0 0 841745271) 0x7f5fa8000b90 con 0x7f5fa403a420 2024-04-27T01:05:45.812 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:05:45.812 INFO:teuthology.orchestra.run.smithi162.stdout:[{"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}, {"addr": "172.21.15.181", "hostname": "smithi181", "labels": [], "status": ""}] 2024-04-27T01:05:45.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.821+0000 7f5fc18ed700 1 -- 172.21.15.162:0/2419474502 >> v1:172.21.15.162:6800/2166195007 conn(0x7f5fa403a420 legacy=0x7f5fa403c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:45.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.821+0000 7f5fc18ed700 1 -- 172.21.15.162:0/2419474502 >> v1:172.21.15.162:6789/0 conn(0x7f5fbc101db0 legacy=0x7f5fbc19c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:45.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.821+0000 7f5fc18ed700 1 -- 172.21.15.162:0/2419474502 shutdown_connections 2024-04-27T01:05:45.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.821+0000 7f5fc18ed700 1 -- 172.21.15.162:0/2419474502 >> 172.21.15.162:0/2419474502 conn(0x7f5fbc0fd660 msgr2=0x7f5fbc108bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:45.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.821+0000 7f5fc18ed700 1 -- 172.21.15.162:0/2419474502 shutdown_connections 2024-04-27T01:05:45.814 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:45.821+0000 7f5fc18ed700 1 -- 172.21.15.162:0/2419474502 wait complete. 2024-04-27T01:05:46.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:45 smithi162 ceph-mon[26714]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:46.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:45 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:46.442 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-27T01:05:46.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-27T01:05:46.678 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:05:47.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:46 smithi162 ceph-mon[26714]: from='client.14180 v1:172.21.15.162:0/2419474502' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:05:47.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.996+0000 7fa03de2b700 1 -- 172.21.15.162:0/852460584 >> v1:172.21.15.162:6789/0 conn(0x7fa038106a50 legacy=0x7fa038106e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:47.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.996+0000 7fa03de2b700 1 -- 172.21.15.162:0/852460584 shutdown_connections 2024-04-27T01:05:47.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.996+0000 7fa03de2b700 1 -- 172.21.15.162:0/852460584 >> 172.21.15.162:0/852460584 conn(0x7fa0380fc580 msgr2=0x7fa0380fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:47.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.996+0000 7fa03de2b700 1 -- 172.21.15.162:0/852460584 shutdown_connections 2024-04-27T01:05:47.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.996+0000 7fa03de2b700 1 -- 172.21.15.162:0/852460584 wait complete. 2024-04-27T01:05:47.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.997+0000 7fa03de2b700 1 Processor -- start 2024-04-27T01:05:47.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.997+0000 7fa03de2b700 1 -- start start 2024-04-27T01:05:47.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.997+0000 7fa03de2b700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fa03819bdb0 con 0x7fa038106a50 2024-04-27T01:05:47.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa0377fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fa038106a50 0x7fa03819b6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:50396/0 (socket says 172.21.15.162:50396) 2024-04-27T01:05:47.991 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa0377fe700 1 -- 172.21.15.162:0/1030802484 learned_addr learned my addr 172.21.15.162:0/1030802484 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063863110 0 0) 0x7fa03819bdb0 con 0x7fa038106a50 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa020005530 con 0x7fa038106a50 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3471109557 0 0) 0x7fa020005530 con 0x7fa038106a50 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa03819bdb0 con 0x7fa038106a50 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fa0280047e0 con 0x7fa038106a50 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 147413398 0 0) 0x7fa03819bdb0 con 0x7fa038106a50 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03819bf80 con 0x7fa038106a50 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa03de2b700 1 -- 172.21.15.162:0/1030802484 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa03819c270 con 0x7fa038106a50 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa03de2b700 1 -- 172.21.15.162:0/1030802484 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa03819e7e0 con 0x7fa038106a50 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fa028001a80 con 0x7fa038106a50 2024-04-27T01:05:47.992 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.998+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7fa028004d30 con 0x7fa038106a50 2024-04-27T01:05:47.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.999+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7fa028004ef0 con 0x7fa038106a50 2024-04-27T01:05:47.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.999+0000 7fa03de2b700 1 -- 172.21.15.162:0/1030802484 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa024005120 con 0x7fa038106a50 2024-04-27T01:05:47.993 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:47.999+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 1069+0+0 (unknown 1300520211 0 0) 0x7fa028047770 con 0x7fa038106a50 2024-04-27T01:05:47.996 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:48.004+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fa0381075d0 con 0x7fa038106a50 2024-04-27T01:05:48.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:47 smithi162 ceph-mon[26714]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:48.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:48.203+0000 7fa03de2b700 1 -- 172.21.15.162:0/1030802484 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fa024005410 con 0x7fa038106a50 2024-04-27T01:05:48.970 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:48.976+0000 7fa034ff9700 1 -- 172.21.15.162:0/1030802484 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (unknown 3126668360 0 0) 0x7fa0280173a0 con 0x7fa038106a50 2024-04-27T01:05:48.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:48.979+0000 7fa03de2b700 1 -- 172.21.15.162:0/1030802484 >> v1:172.21.15.162:6800/2166195007 conn(0x7fa02003a510 legacy=0x7fa02003c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:48.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:48.979+0000 7fa03de2b700 1 -- 172.21.15.162:0/1030802484 >> v1:172.21.15.162:6789/0 conn(0x7fa038106a50 legacy=0x7fa03819b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:48.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:48.980+0000 7fa03de2b700 1 -- 172.21.15.162:0/1030802484 shutdown_connections 2024-04-27T01:05:48.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:48.980+0000 7fa03de2b700 1 -- 172.21.15.162:0/1030802484 >> 172.21.15.162:0/1030802484 conn(0x7fa0380fc580 msgr2=0x7fa0381089c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:48.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:48.980+0000 7fa03de2b700 1 -- 172.21.15.162:0/1030802484 shutdown_connections 2024-04-27T01:05:48.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:05:48.980+0000 7fa03de2b700 1 -- 172.21.15.162:0/1030802484 wait complete. 2024-04-27T01:05:48.973 INFO:teuthology.orchestra.run.smithi162.stderr:adjusted tunables profile to default 2024-04-27T01:05:49.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1030802484' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-27T01:05:49.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:48 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:49.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:48 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:49.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:48 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:49.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:48 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:49.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:48 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:05:49.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:48 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:05:49.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:48 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:05:49.634 INFO:tasks.cephadm:Adding mon.a on smithi162 2024-04-27T01:05:49.634 INFO:tasks.cephadm:Adding mon.c on smithi162 2024-04-27T01:05:49.634 INFO:tasks.cephadm:Adding mon.b on smithi181 2024-04-27T01:05:49.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi162:[v1:172.21.15.162:6789]=a;smithi162:[v1:172.21.15.162:6790]=c;smithi181:[v1:172.21.15.181:6789]=b' 2024-04-27T01:05:49.867 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:05:50.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:49 smithi162 ceph-mon[26714]: Updating smithi181:/etc/ceph/ceph.conf 2024-04-27T01:05:50.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:49 smithi162 ceph-mon[26714]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:50.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1030802484' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-27T01:05:50.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:49 smithi162 ceph-mon[26714]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:05:50.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:49 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:50.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:49 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:50.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:49 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:51.268 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.274+0000 7f81876ec700 1 -- 172.21.15.181:0/3974278181 >> v1:172.21.15.162:6789/0 conn(0x7f8180106a50 legacy=0x7f8180106e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:51.268 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.274+0000 7f81876ec700 1 -- 172.21.15.181:0/3974278181 shutdown_connections 2024-04-27T01:05:51.268 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.274+0000 7f81876ec700 1 -- 172.21.15.181:0/3974278181 >> 172.21.15.181:0/3974278181 conn(0x7f81800fc520 msgr2=0x7f81800fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:51.268 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.274+0000 7f81876ec700 1 -- 172.21.15.181:0/3974278181 shutdown_connections 2024-04-27T01:05:51.268 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.274+0000 7f81876ec700 1 -- 172.21.15.181:0/3974278181 wait complete. 2024-04-27T01:05:51.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.275+0000 7f81876ec700 1 Processor -- start 2024-04-27T01:05:51.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.275+0000 7f81876ec700 1 -- start start 2024-04-27T01:05:51.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.275+0000 7f81876ec700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f818019bdd0 con 0x7f8180106a50 2024-04-27T01:05:51.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.275+0000 7f8185488700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f8180106a50 0x7f818019b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.181:44006/0 (socket says 172.21.15.181:44006) 2024-04-27T01:05:51.269 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.275+0000 7f8185488700 1 -- 172.21.15.181:0/2534845927 learned_addr learned my addr 172.21.15.181:0/2534845927 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:05:51.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.275+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1711006930 0 0) 0x7f818019bdd0 con 0x7f8180106a50 2024-04-27T01:05:51.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.275+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f816c005530 con 0x7f8180106a50 2024-04-27T01:05:51.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.275+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 781646077 0 0) 0x7f816c005530 con 0x7f8180106a50 2024-04-27T01:05:51.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.275+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f818019bdd0 con 0x7f8180106a50 2024-04-27T01:05:51.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.275+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f817c0048b0 con 0x7f8180106a50 2024-04-27T01:05:51.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.276+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3418319148 0 0) 0x7f818019bdd0 con 0x7f8180106a50 2024-04-27T01:05:51.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.276+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818019bfa0 con 0x7f8180106a50 2024-04-27T01:05:51.271 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.276+0000 7f81876ec700 1 -- 172.21.15.181:0/2534845927 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f818019c290 con 0x7f8180106a50 2024-04-27T01:05:51.271 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.276+0000 7f81876ec700 1 -- 172.21.15.181:0/2534845927 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f818019e800 con 0x7f8180106a50 2024-04-27T01:05:51.271 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.276+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f817c003740 con 0x7f8180106a50 2024-04-27T01:05:51.271 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.276+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f817c004d20 con 0x7f8180106a50 2024-04-27T01:05:51.271 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.277+0000 7f81876ec700 1 -- 172.21.15.181:0/2534845927 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8164005120 con 0x7f8180106a50 2024-04-27T01:05:51.271 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.277+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7f817c004ec0 con 0x7f8180106a50 2024-04-27T01:05:51.271 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.277+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1547207639 0 0) 0x7f817c047870 con 0x7f8180106a50 2024-04-27T01:05:51.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.282+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f81801075d0 con 0x7f8180106a50 2024-04-27T01:05:51.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:50 smithi162 ceph-mon[26714]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:05:51.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:50 smithi162 ceph-mon[26714]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:05:51.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:50 smithi162 ceph-mon[26714]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:05:51.480 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.486+0000 7f81876ec700 1 -- 172.21.15.181:0/2534845927 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi162:[v1:172.21.15.162:6789]=a;smithi162:[v1:172.21.15.162:6790]=c;smithi181:[v1:172.21.15.181:6789]=b", "target": ["mon-mgr", ""]}) -- 0x7f8164000c70 con 0x7f816c03a420 2024-04-27T01:05:51.489 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.496+0000 7f81767fc700 1 -- 172.21.15.181:0/2534845927 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 3265049985) 0x7f8164000c70 con 0x7f816c03a420 2024-04-27T01:05:51.489 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mon update... 2024-04-27T01:05:51.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.498+0000 7f81876ec700 1 -- 172.21.15.181:0/2534845927 >> v1:172.21.15.162:6800/2166195007 conn(0x7f816c03a420 legacy=0x7f816c03c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:51.491 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.498+0000 7f81876ec700 1 -- 172.21.15.181:0/2534845927 >> v1:172.21.15.162:6789/0 conn(0x7f8180106a50 legacy=0x7f818019b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:51.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.498+0000 7f81876ec700 1 -- 172.21.15.181:0/2534845927 shutdown_connections 2024-04-27T01:05:51.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.498+0000 7f81876ec700 1 -- 172.21.15.181:0/2534845927 >> 172.21.15.181:0/2534845927 conn(0x7f81800fc520 msgr2=0x7f8180108a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:51.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.498+0000 7f81876ec700 1 -- 172.21.15.181:0/2534845927 shutdown_connections 2024-04-27T01:05:51.492 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:51.498+0000 7f81876ec700 1 -- 172.21.15.181:0/2534845927 wait complete. 2024-04-27T01:05:52.114 DEBUG:teuthology.orchestra.run.smithi162:mon.c> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.c.service 2024-04-27T01:05:52.117 DEBUG:teuthology.orchestra.run.smithi181:mon.b> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.b.service 2024-04-27T01:05:52.120 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-27T01:05:52.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:05:52.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:51 smithi162 ceph-mon[26714]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:52.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:52.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:05:52.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:05:52.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:05:52.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:52.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:05:52.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:05:52.822 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:05:53.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:52 smithi162 ceph-mon[26714]: from='client.14184 v1:172.21.15.181:0/2534845927' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi162:[v1:172.21.15.162:6789]=a;smithi162:[v1:172.21.15.162:6790]=c;smithi181:[v1:172.21.15.181:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:53.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:52 smithi162 ceph-mon[26714]: Saving service mon spec with placement smithi162:[v1:172.21.15.162:6789]=a;smithi162:[v1:172.21.15.162:6790]=c;smithi181:[v1:172.21.15.181:6789]=b;count:3 2024-04-27T01:05:53.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:52 smithi162 ceph-mon[26714]: Deploying daemon mon.b on smithi181 2024-04-27T01:05:54.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:53 smithi162 ceph-mon[26714]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:55.709 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.703+0000 7f7140700700 1 -- 172.21.15.181:0/2062905748 >> v1:172.21.15.162:6789/0 conn(0x7f7138106a80 legacy=0x7f7138106e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:55.709 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.703+0000 7f7140700700 1 -- 172.21.15.181:0/2062905748 shutdown_connections 2024-04-27T01:05:55.709 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.703+0000 7f7140700700 1 -- 172.21.15.181:0/2062905748 >> 172.21.15.181:0/2062905748 conn(0x7f71380fc5d0 msgr2=0x7f71380fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:55.709 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.703+0000 7f7140700700 1 -- 172.21.15.181:0/2062905748 shutdown_connections 2024-04-27T01:05:55.709 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.703+0000 7f7140700700 1 -- 172.21.15.181:0/2062905748 wait complete. 2024-04-27T01:05:55.709 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.704+0000 7f7140700700 1 Processor -- start 2024-04-27T01:05:55.709 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.704+0000 7f7140700700 1 -- start start 2024-04-27T01:05:55.709 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.704+0000 7f7140700700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7138198110 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.704+0000 7f713e49c700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f7138106a80 0x7f7138197a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.181:54940/0 (socket says 172.21.15.181:54940) 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.704+0000 7f713e49c700 1 -- 172.21.15.181:0/3478753213 learned_addr learned my addr 172.21.15.181:0/3478753213 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3351903215 0 0) 0x7f7138198110 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7124005530 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3603870531 0 0) 0x7f7124005530 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7138198110 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f71280047e0 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 450494997 0 0) 0x7f7138198110 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71381982e0 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f7140700700 1 -- 172.21.15.181:0/3478753213 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71381985d0 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f7140700700 1 -- 172.21.15.181:0/3478753213 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f713819fc40 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f7128001a80 con 0x7f7138106a80 2024-04-27T01:05:55.710 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.705+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 170+0+0 (unknown 237500970 0 0) 0x7f7128004dc0 con 0x7f7138106a80 2024-04-27T01:05:55.711 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.706+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7f7128005000 con 0x7f7138106a80 2024-04-27T01:05:55.711 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.706+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1547207639 0 0) 0x7f7128047960 con 0x7f7138106a80 2024-04-27T01:05:55.711 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.706+0000 7f7140700700 1 -- 172.21.15.181:0/3478753213 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f711c005120 con 0x7f7138106a80 2024-04-27T01:05:55.711 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:55.711+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f7138107600 con 0x7f7138106a80 2024-04-27T01:05:56.012 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:56.018+0000 7f7140700700 1 -- 172.21.15.181:0/3478753213 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f711c005410 con 0x7f7138106a80 2024-04-27T01:05:56.012 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:56.019+0000 7f712f7fe700 1 -- 172.21.15.181:0/3478753213 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+697 (unknown 2237029548 0 3848881819) 0x7f7128017600 con 0x7f7138106a80 2024-04-27T01:05:56.012 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:05:56.013 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"ea378f3c-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T01:04:16.109272Z","created":"2024-04-27T01:04:16.109272Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-27T01:05:56.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:56.021+0000 7f7140700700 1 -- 172.21.15.181:0/3478753213 >> v1:172.21.15.162:6800/2166195007 conn(0x7f712403a420 legacy=0x7f712403c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:56.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:56.021+0000 7f7140700700 1 -- 172.21.15.181:0/3478753213 >> v1:172.21.15.162:6789/0 conn(0x7f7138106a80 legacy=0x7f7138197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:05:56.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:56.021+0000 7f7140700700 1 -- 172.21.15.181:0/3478753213 shutdown_connections 2024-04-27T01:05:56.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:56.021+0000 7f7140700700 1 -- 172.21.15.181:0/3478753213 >> 172.21.15.181:0/3478753213 conn(0x7f71380fc5d0 msgr2=0x7f71380fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:05:56.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:56.021+0000 7f7140700700 1 -- 172.21.15.181:0/3478753213 shutdown_connections 2024-04-27T01:05:56.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:05:56.021+0000 7f7140700700 1 -- 172.21.15.181:0/3478753213 wait complete. 2024-04-27T01:05:56.015 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2024-04-27T01:05:56.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:56 smithi181 systemd[1]: Starting Ceph mon.b for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:05:56.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:55 smithi162 ceph-mon[26714]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:57.115 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:56 smithi181 podman[30217]: 2024-04-27 01:05:56.754873903 +0000 UTC m=+0.017940699 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:05:57.115 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:56 smithi181 podman[30217]: 2024-04-27 01:05:56.854054968 +0000 UTC m=+0.117121746 container create f4dafb30ceb6b30ebdfdf89f047e06a20400f49b8ed47abc38df32fbf3024d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-b, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 podman[30217]: 2024-04-27 01:05:57.047954208 +0000 UTC m=+0.311020996 container init f4dafb30ceb6b30ebdfdf89f047e06a20400f49b8ed47abc38df32fbf3024d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 podman[30217]: 2024-04-27 01:05:57.050919254 +0000 UTC m=+0.313986033 container start f4dafb30ceb6b30ebdfdf89f047e06a20400f49b8ed47abc38df32fbf3024d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-b, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev), process ceph-mon, pid 2 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: pidfile_write: ignore empty --pid-file 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: load: jerasure load: lrc 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: RocksDB version: 7.9.2 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Git sha 0 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Compile date 2024-04-25 16:43:49 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: DB SUMMARY 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: DB Session ID: 679WCBFYCZNHZQFG9R41 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: CURRENT file: CURRENT 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T01:05:57.116 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.error_if_exists: 0 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.create_if_missing: 0 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.paranoid_checks: 1 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.env: 0x5624a01cfc60 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.fs: PosixFileSystem 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.info_log: 0x5624a0fabb00 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.statistics: (nil) 2024-04-27T01:05:57.117 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.use_fsync: 0 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_log_file_size: 0 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.allow_fallocate: 1 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.use_direct_reads: 0 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.db_log_dir: 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.wal_dir: 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T01:05:57.118 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.write_buffer_manager: 0x5624a0f7e1e0 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T01:05:57.119 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T01:05:57.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T01:05:57.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T01:05:57.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T01:05:57.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T01:05:57.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.unordered_write: 0 2024-04-27T01:05:57.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T01:05:57.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T01:05:57.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T01:05:57.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T01:05:57.121 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.row_cache: None 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.wal_filter: None 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.two_write_queues: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.wal_compression: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.atomic_flush: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.log_readahead_size: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T01:05:57.122 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_background_jobs: 2 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_background_compactions: -1 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_subcompactions: 1 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T01:05:57.123 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_open_files: -1 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_background_flushes: -1 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Compression algorithms supported: 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: kZSTD supported: 0 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: kXpressCompression supported: 0 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: kZlibCompression supported: 1 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: kSnappyCompression supported: 1 2024-04-27T01:05:57.124 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: kLZ4Compression supported: 1 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: kBZip2Compression supported: 0 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.merge_operator: 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_filter: None 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5624a0fabc20) 2024-04-27T01:05:57.125 INFO:journalctl@ceph.mon.b.smithi181.stdout: cache_index_and_filter_blocks: 1 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: pin_top_level_index_and_filter: 1 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: index_type: 0 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: data_block_index_type: 0 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: index_shortening: 1 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: checksum: 4 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: no_block_cache: 0 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: block_cache: 0x5624a0faf090 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: block_cache_name: BinnedLRUCache 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: block_cache_options: 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: capacity : 536870912 2024-04-27T01:05:57.126 INFO:journalctl@ceph.mon.b.smithi181.stdout: num_shard_bits : 4 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: strict_capacity_limit : 0 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: high_pri_pool_ratio: 0.000 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: block_cache_compressed: (nil) 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: persistent_cache: (nil) 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: block_size: 4096 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: block_size_deviation: 10 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: block_restart_interval: 16 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: index_block_restart_interval: 1 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: metadata_block_size: 4096 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: partition_filters: 0 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: use_delta_encoding: 1 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: filter_policy: bloomfilter 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: whole_key_filtering: 1 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: verify_compression: 0 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: read_amp_bytes_per_bit: 0 2024-04-27T01:05:57.127 INFO:journalctl@ceph.mon.b.smithi181.stdout: format_version: 5 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout: enable_index_compression: 1 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout: block_align: 0 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout: max_auto_readahead_size: 262144 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout: prepopulate_block_cache: 0 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout: initial_auto_readahead_size: 8192 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout: num_file_reads_for_auto_readahead: 2 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compression: NoCompression 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.num_levels: 7 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T01:05:57.128 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:05:57.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T01:05:57.130 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.arena_block_size: 1048576 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T01:05:57.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.table_properties_collectors: 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.inplace_update_support: 0 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.bloom_locality: 0 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.max_successive_merges: 0 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T01:05:57.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.ttl: 2592000 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.enable_blob_files: false 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.min_blob_size: 0 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.blob_file_starting_level: 0 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-27T01:05:57.133 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 15b643a3-b764-4081-a7db-3c4456c87f4c 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179957080876, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179957081452, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1587, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714179957, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "15b643a3-b764-4081-a7db-3c4456c87f4c", "db_session_id": "679WCBFYCZNHZQFG9R41", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179957081563, "job": 1, "event": "recovery_finished"} 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5624a10a0000 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: DB pointer 0x5624a108a000 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-27T01:05:57.134 INFO:journalctl@ceph.mon.b.smithi181.stdout: ** DB Stats ** 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: ** Compaction Stats [default] ** 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: L0 1/0 1.55 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: Sum 1/0 1.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: 2024-04-27T01:05:57.135 INFO:journalctl@ceph.mon.b.smithi181.stdout: ** Compaction Stats [default] ** 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: Cumulative compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:05:57.136 INFO:journalctl@ceph.mon.b.smithi181.stdout: Interval compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: Block cache BinnedLRUCache@0x5624a0faf090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: using public_addrv v1:172.21.15.181:6789/0 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: starting mon.b rank -1 at public addrs v1:172.21.15.181:6789/0 at bind addrs v1:172.21.15.181:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(???) e0 preinit fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).mds e1 new map 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: e1 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: legacy client fscid: -1 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: 2024-04-27T01:05:57.137 INFO:journalctl@ceph.mon.b.smithi181.stdout: No filesystems configured 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: Added host smithi181 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.138 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='client.14180 v1:172.21.15.162:0/2419474502' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1030802484' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: Updating smithi181:/etc/ceph/ceph.conf 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1030802484' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.139 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='client.14184 v1:172.21.15.181:0/2534845927' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi162:[v1:172.21.15.162:6789]=a;smithi162:[v1:172.21.15.162:6790]=c;smithi181:[v1:172.21.15.181:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: Saving service mon spec with placement smithi162:[v1:172.21.15.162:6789]=a;smithi162:[v1:172.21.15.162:6790]=c;smithi181:[v1:172.21.15.181:6789]=b;count:3 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: Deploying daemon mon.b on smithi181 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:57.140 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:05:57.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/3478753213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:05:57.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 ceph-mon[30237]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-27T01:05:57.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 bash[30217]: f4dafb30ceb6b30ebdfdf89f047e06a20400f49b8ed47abc38df32fbf3024d4a 2024-04-27T01:05:57.141 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:05:57 smithi181 systemd[1]: Started Ceph mon.b for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:05:57.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:05:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/3478753213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:05:57.615 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-27T01:05:57.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:05:57.857 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:06:02.115 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.122+0000 7f57b61a9700 1 -- 172.21.15.181:0/827757961 >> v1:172.21.15.162:6789/0 conn(0x7f57b0100570 legacy=0x7f57b0100950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:02.115 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.122+0000 7f57b61a9700 1 -- 172.21.15.181:0/827757961 shutdown_connections 2024-04-27T01:06:02.115 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.122+0000 7f57b61a9700 1 -- 172.21.15.181:0/827757961 >> 172.21.15.181:0/827757961 conn(0x7f57b0077320 msgr2=0x7f57b0077730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:02.115 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.122+0000 7f57b61a9700 1 -- 172.21.15.181:0/827757961 shutdown_connections 2024-04-27T01:06:02.116 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.122+0000 7f57b61a9700 1 -- 172.21.15.181:0/827757961 wait complete. 2024-04-27T01:06:02.116 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.123+0000 7f57b61a9700 1 Processor -- start 2024-04-27T01:06:02.116 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.123+0000 7f57b61a9700 1 -- start start 2024-04-27T01:06:02.116 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.124+0000 7f57b61a9700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57b01a4f00 con 0x7f57b0100570 2024-04-27T01:06:02.116 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.124+0000 7f57b61a9700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f57b01a60e0 con 0x7f57b019f150 2024-04-27T01:06:02.116 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.124+0000 7f57aeffd700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f57b019f150 0x7f57b01a37e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:33032/0 (socket says 172.21.15.181:33032) 2024-04-27T01:06:02.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.124+0000 7f57aeffd700 1 -- 172.21.15.181:0/92088694 learned_addr learned my addr 172.21.15.181:0/92088694 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:06:02.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.124+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652116478 0 0) 0x7f57b01a4f00 con 0x7f57b0100570 2024-04-27T01:06:02.117 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.124+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57980046c0 con 0x7f57b0100570 2024-04-27T01:06:02.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.125+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 714824823 0 0) 0x7f57980046c0 con 0x7f57b0100570 2024-04-27T01:06:02.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.125+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f57b01a4f00 con 0x7f57b0100570 2024-04-27T01:06:02.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.125+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 239+0+0 (unknown 3584820739 0 0) 0x7f57a0001500 con 0x7f57b0100570 2024-04-27T01:06:02.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.125+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3612624423 0 0) 0x7f57b01a4f00 con 0x7f57b0100570 2024-04-27T01:06:02.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.125+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 >> v1:172.21.15.181:6789/0 conn(0x7f57b019f150 legacy=0x7f57b01a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:02.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.125+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b01a72c0 con 0x7f57b0100570 2024-04-27T01:06:02.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.125+0000 7f57b61a9700 1 -- 172.21.15.181:0/92088694 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f57b01a5130 con 0x7f57b0100570 2024-04-27T01:06:02.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.125+0000 7f57b61a9700 1 -- 172.21.15.181:0/92088694 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f57b01a5630 con 0x7f57b0100570 2024-04-27T01:06:02.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.126+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f57a0003a10 con 0x7f57b0100570 2024-04-27T01:06:02.118 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.126+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 239+0+0 (unknown 3584820739 0 0) 0x7f57a0004be0 con 0x7f57b0100570 2024-04-27T01:06:02.119 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.127+0000 7f57b61a9700 1 -- 172.21.15.181:0/92088694 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f579c005120 con 0x7f57b0100570 2024-04-27T01:06:02.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.128+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7f57a00017a0 con 0x7f57b0100570 2024-04-27T01:06:02.120 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.128+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1547207639 0 0) 0x7f57a00466c0 con 0x7f57b0100570 2024-04-27T01:06:02.127 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.135+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f57a00153b0 con 0x7f57b0100570 2024-04-27T01:06:02.408 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.415+0000 7f57b61a9700 1 -- 172.21.15.181:0/92088694 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f579c005410 con 0x7f57b0100570 2024-04-27T01:06:02.408 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:02.408 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:06:02.408 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: mon.a calling monitor election 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: mon.b calling monitor election 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: monmap epoch 2 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: last_changed 2024-04-27T01:05:57.102708+0000 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: created 2024-04-27T01:04:16.109272+0000 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: min_mon_release 19 (squid) 2024-04-27T01:06:02.409 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: election_strategy: 1 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: 0: v1:172.21.15.162:6789/0 mon.a 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: 1: v1:172.21.15.181:6789/0 mon.b 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: fsmap 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: mgrmap e13: y(active, since 41s) 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: overall HEALTH_OK 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:02.410 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:02.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.416+0000 7f57acff9700 1 -- 172.21.15.181:0/92088694 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+915 (unknown 3557084514 0 2388100970) 0x7f57a0014cb0 con 0x7f57b0100570 2024-04-27T01:06:02.411 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:06:02.411 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"ea378f3c-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T01:05:57.102708Z","created":"2024-04-27T01:04:16.109272Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-27T01:06:02.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.418+0000 7f57b61a9700 1 -- 172.21.15.181:0/92088694 >> v1:172.21.15.162:6800/2166195007 conn(0x7f579803a4f0 legacy=0x7f579803c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:02.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.418+0000 7f57b61a9700 1 -- 172.21.15.181:0/92088694 >> v1:172.21.15.162:6789/0 conn(0x7f57b0100570 legacy=0x7f57b019ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:02.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.418+0000 7f57b61a9700 1 -- 172.21.15.181:0/92088694 shutdown_connections 2024-04-27T01:06:02.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.418+0000 7f57b61a9700 1 -- 172.21.15.181:0/92088694 >> 172.21.15.181:0/92088694 conn(0x7f57b0077320 msgr2=0x7f57b0107dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:02.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.418+0000 7f57b61a9700 1 -- 172.21.15.181:0/92088694 shutdown_connections 2024-04-27T01:06:02.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:02.418+0000 7f57b61a9700 1 -- 172.21.15.181:0/92088694 wait complete. 2024-04-27T01:06:02.412 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2024-04-27T01:06:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: mon.a calling monitor election 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: mon.b calling monitor election 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: monmap epoch 2 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:06:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: last_changed 2024-04-27T01:05:57.102708+0000 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: created 2024-04-27T01:04:16.109272+0000 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: min_mon_release 19 (squid) 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: election_strategy: 1 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: 0: v1:172.21.15.162:6789/0 mon.a 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: 1: v1:172.21.15.181:6789/0 mon.b 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: fsmap 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: mgrmap e13: y(active, since 41s) 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: overall HEALTH_OK 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:02.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:03.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:03 smithi181 ceph-mon[30237]: Deploying daemon mon.c on smithi162 2024-04-27T01:06:03.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/92088694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:06:03.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:03 smithi181 ceph-mon[30237]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:03.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:03 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:03.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:03 smithi162 ceph-mon[26714]: Deploying daemon mon.c on smithi162 2024-04-27T01:06:03.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/92088694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:06:03.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:03 smithi162 ceph-mon[26714]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:03.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:03 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:03.990 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-27T01:06:03.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-27T01:06:04.232 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:06:05.270 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 systemd[1]: Starting Ceph mon.c for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:06:05.521 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 podman[37661]: 2024-04-27 01:06:05.181588074 +0000 UTC m=+0.018569233 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:06:05.521 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 podman[37661]: 2024-04-27 01:06:05.286055768 +0000 UTC m=+0.123036928 container create d8b6557e66817d0261f49cb8f1953d000782e700024885e6d8d9e8a6289dbbb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-27T01:06:05.521 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 podman[37661]: 2024-04-27 01:06:05.496892275 +0000 UTC m=+0.333873436 container init d8b6557e66817d0261f49cb8f1953d000782e700024885e6d8d9e8a6289dbbb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-27T01:06:05.521 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 podman[37661]: 2024-04-27 01:06:05.500009417 +0000 UTC m=+0.336990578 container start d8b6557e66817d0261f49cb8f1953d000782e700024885e6d8d9e8a6289dbbb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-27T01:06:05.521 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: set uid:gid to 167:167 (ceph:ceph) 2024-04-27T01:06:05.521 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: ceph version 19.0.0-2447-gb22e2ebd (b22e2ebdeb24376882b7bda2a7329c8cccc2276a) squid (dev), process ceph-mon, pid 2 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pidfile_write: ignore empty --pid-file 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: load: jerasure load: lrc 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: RocksDB version: 7.9.2 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Git sha 0 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Compile date 2024-04-25 16:43:49 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: DB SUMMARY 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: DB Session ID: RO7QBXN03XY3BQTSD2NC 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: CURRENT file: CURRENT 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: IDENTITY file: IDENTITY 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-27T01:06:05.522 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-04-27T01:06:05.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.error_if_exists: 0 2024-04-27T01:06:05.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.create_if_missing: 0 2024-04-27T01:06:05.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.paranoid_checks: 1 2024-04-27T01:06:05.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-27T01:06:05.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-27T01:06:05.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-27T01:06:05.523 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.env: 0x5632107a1c60 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.fs: PosixFileSystem 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.info_log: 0x563212d41ac0 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_file_opening_threads: 16 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.statistics: (nil) 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.use_fsync: 0 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_log_file_size: 0 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.keep_log_file_num: 1000 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.recycle_log_file_num: 0 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.allow_fallocate: 1 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.allow_mmap_reads: 0 2024-04-27T01:06:05.524 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.allow_mmap_writes: 0 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.use_direct_reads: 0 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.create_missing_column_families: 0 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.db_log_dir: 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.wal_dir: 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.advise_random_on_open: 1 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.db_write_buffer_size: 0 2024-04-27T01:06:05.525 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.write_buffer_manager: 0x563212d141e0 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.rate_limiter: (nil) 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.wal_recovery_mode: 2 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.enable_thread_tracking: 0 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.enable_pipelined_write: 0 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.unordered_write: 0 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-27T01:06:05.526 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.row_cache: None 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.wal_filter: None 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.allow_ingest_behind: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.two_write_queues: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.manual_wal_flush: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.wal_compression: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.atomic_flush: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.log_readahead_size: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.best_efforts_recovery: 0 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-27T01:06:05.527 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.allow_data_in_errors: 0 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.db_host_id: __hostname__ 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_background_jobs: 2 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_background_compactions: -1 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_subcompactions: 1 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_total_wal_size: 0 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_open_files: -1 2024-04-27T01:06:05.528 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bytes_per_sync: 0 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_readahead_size: 0 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_background_flushes: -1 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Compression algorithms supported: 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: kZSTD supported: 0 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: kXpressCompression supported: 0 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: kLZ4HCCompression supported: 1 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: kZlibCompression supported: 1 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: kSnappyCompression supported: 1 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: kLZ4Compression supported: 1 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: kBZip2Compression supported: 0 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-27T01:06:05.529 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.merge_operator: 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_filter: None 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_filter_factory: None 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.sst_partitioner_factory: None 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563212d41bc0) 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout: cache_index_and_filter_blocks: 1 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout: pin_top_level_index_and_filter: 1 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_type: 0 2024-04-27T01:06:05.530 INFO:journalctl@ceph.mon.c.smithi162.stdout: data_block_index_type: 0 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_shortening: 1 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: checksum: 4 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: no_block_cache: 0 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache: 0x563212d45090 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_name: BinnedLRUCache 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_options: 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: capacity : 536870912 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: num_shard_bits : 4 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: strict_capacity_limit : 0 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: high_pri_pool_ratio: 0.000 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_compressed: (nil) 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: persistent_cache: (nil) 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_size: 4096 2024-04-27T01:06:05.531 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_size_deviation: 10 2024-04-27T01:06:05.532 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_restart_interval: 16 2024-04-27T01:06:05.532 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_block_restart_interval: 1 2024-04-27T01:06:05.532 INFO:journalctl@ceph.mon.c.smithi162.stdout: metadata_block_size: 4096 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: partition_filters: 0 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: use_delta_encoding: 1 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: filter_policy: bloomfilter 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: whole_key_filtering: 1 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: verify_compression: 0 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: read_amp_bytes_per_bit: 0 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: format_version: 5 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: enable_index_compression: 1 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_align: 0 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: max_auto_readahead_size: 262144 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: prepopulate_block_cache: 0 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: initial_auto_readahead_size: 8192 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout: num_file_reads_for_auto_readahead: 2 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.write_buffer_size: 33554432 2024-04-27T01:06:05.533 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_write_buffer_number: 2 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compression: NoCompression 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bottommost_compression: Disabled 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.prefix_extractor: nullptr 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.num_levels: 7 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:06:05.534 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compression_opts.level: 32767 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compression_opts.strategy: 0 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compression_opts.enabled: false 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-27T01:06:05.535 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.target_file_size_base: 67108864 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-27T01:06:05.536 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.arena_block_size: 1048576 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.disable_auto_compactions: 0 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-27T01:06:05.537 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.table_properties_collectors: 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.inplace_update_support: 0 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.bloom_locality: 0 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.max_successive_merges: 0 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.paranoid_file_checks: 0 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.force_consistency_checks: 1 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.report_bg_io_stats: 0 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.ttl: 2592000 2024-04-27T01:06:05.538 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.enable_blob_files: false 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.min_blob_size: 0 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.blob_file_size: 268435456 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.blob_file_starting_level: 0 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-27T01:06:05.539 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 81a9c796-885a-4b32-8f14-39a6303a39db 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179965529758, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179965530289, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1677, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714179965, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "81a9c796-885a-4b32-8f14-39a6303a39db", "db_session_id": "RO7QBXN03XY3BQTSD2NC", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714179965530396, "job": 1, "event": "recovery_finished"} 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563212e36000 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: DB pointer 0x563212e20000 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** DB Stats ** 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-27T01:06:05.540 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** Compaction Stats [default] ** 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: L0 1/0 1.64 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: Sum 1/0 1.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** Compaction Stats [default] ** 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-27T01:06:05.541 INFO:journalctl@ceph.mon.c.smithi162.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: Block cache BinnedLRUCache@0x563212d45090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-27T01:06:05.542 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-04-27T01:06:05.543 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-27T01:06:05.543 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-27T01:06:05.543 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: using public_addrv v1:172.21.15.162:6790/0 2024-04-27T01:06:05.543 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: starting mon.c rank -1 at public addrs v1:172.21.15.162:6790/0 at bind addrs v1:172.21.15.162:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:06:05.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(???) e0 preinit fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:06:05.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).mds e1 new map 2024-04-27T01:06:05.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-27T01:06:05.842 INFO:journalctl@ceph.mon.c.smithi162.stdout: e1 2024-04-27T01:06:05.842 INFO:journalctl@ceph.mon.c.smithi162.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-27T01:06:05.842 INFO:journalctl@ceph.mon.c.smithi162.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-27T01:06:05.842 INFO:journalctl@ceph.mon.c.smithi162.stdout: legacy client fscid: -1 2024-04-27T01:06:05.842 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-04-27T01:06:05.842 INFO:journalctl@ceph.mon.c.smithi162.stdout: No filesystems configured 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: Added host smithi181 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='client.14180 v1:172.21.15.162:0/2419474502' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1030802484' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:05.844 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:05.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: Updating smithi181:/etc/ceph/ceph.conf 2024-04-27T01:06:05.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:05.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1030802484' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-27T01:06:05.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:06:05.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.845 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:06:05.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:06:05.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:06:05.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:05.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:05.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:05.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:05.846 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='client.14184 v1:172.21.15.181:0/2534845927' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi162:[v1:172.21.15.162:6789]=a;smithi162:[v1:172.21.15.162:6790]=c;smithi181:[v1:172.21.15.181:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: Saving service mon spec with placement smithi162:[v1:172.21.15.162:6789]=a;smithi162:[v1:172.21.15.162:6790]=c;smithi181:[v1:172.21.15.181:6789]=b;count:3 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: Deploying daemon mon.b on smithi181 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/3478753213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:05.847 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.a calling monitor election 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.b calling monitor election 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: monmap epoch 2 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: last_changed 2024-04-27T01:05:57.102708+0000 2024-04-27T01:06:05.848 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: created 2024-04-27T01:04:16.109272+0000 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: min_mon_release 19 (squid) 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: election_strategy: 1 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: 0: v1:172.21.15.162:6789/0 mon.a 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: 1: v1:172.21.15.181:6789/0 mon.b 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: fsmap 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mgrmap e13: y(active, since 41s) 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: overall HEALTH_OK 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:05.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:05.850 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: Deploying daemon mon.c on smithi162 2024-04-27T01:06:05.850 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/92088694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:06:05.850 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:05.850 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:05.850 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 ceph-mon[37680]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-04-27T01:06:05.850 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 bash[37661]: d8b6557e66817d0261f49cb8f1953d000782e700024885e6d8d9e8a6289dbbb4 2024-04-27T01:06:05.850 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:05 smithi162 systemd[1]: Started Ceph mon.c for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:06:10.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.568+0000 7eff17256700 1 -- 172.21.15.181:0/166877607 >> v1:172.21.15.162:6789/0 conn(0x7efef40035f0 legacy=0x7efef4005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:10.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.568+0000 7eff17256700 1 -- 172.21.15.181:0/166877607 shutdown_connections 2024-04-27T01:06:10.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.568+0000 7eff17256700 1 -- 172.21.15.181:0/166877607 >> 172.21.15.181:0/166877607 conn(0x7eff100fc6c0 msgr2=0x7eff100feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:10.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.568+0000 7eff17256700 1 -- 172.21.15.181:0/166877607 shutdown_connections 2024-04-27T01:06:10.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.568+0000 7eff17256700 1 -- 172.21.15.181:0/166877607 wait complete. 2024-04-27T01:06:10.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.569+0000 7eff17256700 1 Processor -- start 2024-04-27T01:06:10.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.569+0000 7eff17256700 1 -- start start 2024-04-27T01:06:10.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.569+0000 7eff17256700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff101a6a40 con 0x7eff10197f90 2024-04-27T01:06:10.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.569+0000 7eff17256700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff101a7c60 con 0x7efef40035f0 2024-04-27T01:06:10.562 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.569+0000 7eff17256700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff101a8ea0 con 0x7eff101984b0 2024-04-27T01:06:10.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.570+0000 7eff14ff2700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7efef40035f0 0x7eff10197880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:33044/0 (socket says 172.21.15.181:33044) 2024-04-27T01:06:10.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.570+0000 7eff14ff2700 1 -- 172.21.15.181:0/3206537745 learned_addr learned my addr 172.21.15.181:0/3206537745 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:06:10.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.570+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3508872242 0 0) 0x7eff101a6a40 con 0x7eff10197f90 2024-04-27T01:06:10.563 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.570+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff00003bf0 con 0x7eff10197f90 2024-04-27T01:06:10.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.571+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1345327649 0 0) 0x7eff00003bf0 con 0x7eff10197f90 2024-04-27T01:06:10.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.571+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff101a6a40 con 0x7eff10197f90 2024-04-27T01:06:10.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.571+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7efefc001830 con 0x7eff10197f90 2024-04-27T01:06:10.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.571+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3942877224 0 0) 0x7eff101a6a40 con 0x7eff10197f90 2024-04-27T01:06:10.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.571+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 >> v1:172.21.15.162:6790/0 conn(0x7eff101984b0 legacy=0x7eff101a5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:10.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.571+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 >> v1:172.21.15.181:6789/0 conn(0x7efef40035f0 legacy=0x7eff10197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:10.564 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.571+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff101aa0e0 con 0x7eff10197f90 2024-04-27T01:06:10.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.571+0000 7eff17256700 1 -- 172.21.15.181:0/3206537745 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff101a90d0 con 0x7eff10197f90 2024-04-27T01:06:10.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.571+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7efefc003b10 con 0x7eff10197f90 2024-04-27T01:06:10.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.571+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7efefc004aa0 con 0x7eff10197f90 2024-04-27T01:06:10.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.572+0000 7eff17256700 1 -- 172.21.15.181:0/3206537745 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7eff101a95d0 con 0x7eff10197f90 2024-04-27T01:06:10.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.572+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7efefc0102a0 con 0x7eff10197f90 2024-04-27T01:06:10.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.573+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1547207639 0 0) 0x7efefc0474c0 con 0x7eff10197f90 2024-04-27T01:06:10.565 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.573+0000 7eff17256700 1 -- 172.21.15.181:0/3206537745 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff101022f0 con 0x7eff10197f90 2024-04-27T01:06:10.575 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.583+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7efefc016210 con 0x7eff10197f90 2024-04-27T01:06:10.862 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.870+0000 7eff17256700 1 -- 172.21.15.181:0/3206537745 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7eff1006aa60 con 0x7eff10197f90 2024-04-27T01:06:10.863 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.870+0000 7eff0dffb700 1 -- 172.21.15.181:0/3206537745 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1133 (unknown 409899127 0 3593125449) 0x7efefc015b10 con 0x7eff10197f90 2024-04-27T01:06:10.863 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-27T01:06:10.863 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":3,"fsid":"ea378f3c-0431-11ef-bc93-c7b262605968","modified":"2024-04-27T01:06:05.550483Z","created":"2024-04-27T01:04:16.109272Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6790","nonce":0}]},"addr":"172.21.15.162:6790/0","public_addr":"172.21.15.162:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-27T01:06:10.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:06:10.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: mon.a calling monitor election 2024-04-27T01:06:10.863 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: mon.b calling monitor election 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: mon.c calling monitor election 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: monmap epoch 3 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: last_changed 2024-04-27T01:06:05.550483+0000 2024-04-27T01:06:10.864 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: created 2024-04-27T01:04:16.109272+0000 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: min_mon_release 19 (squid) 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: election_strategy: 1 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: 0: v1:172.21.15.162:6789/0 mon.a 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: 1: v1:172.21.15.181:6789/0 mon.b 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: 2: v1:172.21.15.162:6790/0 mon.c 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: fsmap 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: mgrmap e13: y(active, since 49s) 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: overall HEALTH_OK 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:10.865 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:10 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:10.866 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.873+0000 7eff17256700 1 -- 172.21.15.181:0/3206537745 >> v1:172.21.15.162:6800/2166195007 conn(0x7eff0003a2c0 legacy=0x7eff0003c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:10.866 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.873+0000 7eff17256700 1 -- 172.21.15.181:0/3206537745 >> v1:172.21.15.162:6789/0 conn(0x7eff10197f90 legacy=0x7eff101a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:10.866 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.873+0000 7eff17256700 1 -- 172.21.15.181:0/3206537745 shutdown_connections 2024-04-27T01:06:10.866 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.873+0000 7eff17256700 1 -- 172.21.15.181:0/3206537745 >> 172.21.15.181:0/3206537745 conn(0x7eff100fc6c0 msgr2=0x7eff100fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:10.867 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.873+0000 7eff17256700 1 -- 172.21.15.181:0/3206537745 shutdown_connections 2024-04-27T01:06:10.867 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:10.873+0000 7eff17256700 1 -- 172.21.15.181:0/3206537745 wait complete. 2024-04-27T01:06:10.867 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 3 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: mon.a calling monitor election 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: mon.b calling monitor election 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: mon.c calling monitor election 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:11.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: monmap epoch 3 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: last_changed 2024-04-27T01:06:05.550483+0000 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: created 2024-04-27T01:04:16.109272+0000 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: min_mon_release 19 (squid) 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: election_strategy: 1 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: 0: v1:172.21.15.162:6789/0 mon.a 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: 1: v1:172.21.15.181:6789/0 mon.b 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: 2: v1:172.21.15.162:6790/0 mon.c 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: fsmap 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:06:11.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: mgrmap e13: y(active, since 49s) 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: overall HEALTH_OK 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: mon.a calling monitor election 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: mon.b calling monitor election 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:11.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: mon.c calling monitor election 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: monmap epoch 3 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: fsid ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: last_changed 2024-04-27T01:06:05.550483+0000 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: created 2024-04-27T01:04:16.109272+0000 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: min_mon_release 19 (squid) 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: election_strategy: 1 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: 0: v1:172.21.15.162:6789/0 mon.a 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: 1: v1:172.21.15.181:6789/0 mon.b 2024-04-27T01:06:11.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: 2: v1:172.21.15.162:6790/0 mon.c 2024-04-27T01:06:11.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: fsmap 2024-04-27T01:06:11.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: osdmap e4: 0 total, 0 up, 0 in 2024-04-27T01:06:11.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: mgrmap e13: y(active, since 49s) 2024-04-27T01:06:11.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: overall HEALTH_OK 2024-04-27T01:06:11.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:10 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.497 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-27T01:06:11.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-27T01:06:11.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:11.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/3206537745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:06:11.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:11 smithi181 ceph-mon[30237]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:11.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:11.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/3206537745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:06:11.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:11 smithi162 ceph-mon[26714]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:11.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:11.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/3206537745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-27T01:06:11.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:11 smithi162 ceph-mon[37680]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:11.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:11.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:06:11.971 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:06:12.783 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:06:12 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:06:12.548+0000 7ff30af11700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-27T01:06:13.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:13.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:06:13.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:13.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.515 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.515 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.515 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:13.515 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:13.515 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:06:13.515 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.603+0000 7ff6e0775700 1 -- 172.21.15.162:0/1070278266 >> v1:172.21.15.162:6789/0 conn(0x7ff6d8106a60 legacy=0x7ff6d8106e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.603+0000 7ff6e0775700 1 -- 172.21.15.162:0/1070278266 shutdown_connections 2024-04-27T01:06:13.597 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.603+0000 7ff6e0775700 1 -- 172.21.15.162:0/1070278266 >> 172.21.15.162:0/1070278266 conn(0x7ff6d80765c0 msgr2=0x7ff6d80769d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.603+0000 7ff6e0775700 1 -- 172.21.15.162:0/1070278266 shutdown_connections 2024-04-27T01:06:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.603+0000 7ff6e0775700 1 -- 172.21.15.162:0/1070278266 wait complete. 2024-04-27T01:06:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6e0775700 1 Processor -- start 2024-04-27T01:06:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6e0775700 1 -- start start 2024-04-27T01:06:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6e0775700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6d81a5b80 con 0x7ff6d81a1e70 2024-04-27T01:06:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6e0775700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6d81a6dc0 con 0x7ff6d8106a60 2024-04-27T01:06:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6e0775700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff6d81a8000 con 0x7ff6d819bfa0 2024-04-27T01:06:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6ded12700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7ff6d81a1e70 0x7ff6d81a4280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:56166/0 (socket says 172.21.15.162:56166) 2024-04-27T01:06:13.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6ded12700 1 -- 172.21.15.162:0/805887509 learned_addr learned my addr 172.21.15.162:0/805887509 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:06:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3905673581 0 0) 0x7ff6d81a5b80 con 0x7ff6d81a1e70 2024-04-27T01:06:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c4003bf0 con 0x7ff6d81a1e70 2024-04-27T01:06:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 614090617 0 0) 0x7ff6d81a8000 con 0x7ff6d819bfa0 2024-04-27T01:06:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d81a5b80 con 0x7ff6d819bfa0 2024-04-27T01:06:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4156986975 0 0) 0x7ff6d81a6dc0 con 0x7ff6d8106a60 2024-04-27T01:06:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.604+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6d81a8000 con 0x7ff6d8106a60 2024-04-27T01:06:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1882160021 0 0) 0x7ff6d81a5b80 con 0x7ff6d819bfa0 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6d81a6dc0 con 0x7ff6d819bfa0 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1466711691 0 0) 0x7ff6c4003bf0 con 0x7ff6d81a1e70 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6d81a5b80 con 0x7ff6d81a1e70 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7ff6d4001810 con 0x7ff6d819bfa0 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7ff6d0001af0 con 0x7ff6d81a1e70 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3883967729 0 0) 0x7ff6d81a8000 con 0x7ff6d8106a60 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6c4003bf0 con 0x7ff6d8106a60 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7ff6c80048d0 con 0x7ff6d8106a60 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1049508834 0 0) 0x7ff6d81a6dc0 con 0x7ff6d819bfa0 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 >> v1:172.21.15.181:6789/0 conn(0x7ff6d8106a60 legacy=0x7ff6d819b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 >> v1:172.21.15.162:6789/0 conn(0x7ff6d81a1e70 legacy=0x7ff6d81a4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d81a9240 con 0x7ff6d819bfa0 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6e0775700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d81a8230 con 0x7ff6d819bfa0 2024-04-27T01:06:13.601 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6e0775700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6d81a86f0 con 0x7ff6d819bfa0 2024-04-27T01:06:13.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7ff6d4003b30 con 0x7ff6d819bfa0 2024-04-27T01:06:13.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.605+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7ff6d4004a80 con 0x7ff6d819bfa0 2024-04-27T01:06:13.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.606+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7ff6d400ff70 con 0x7ff6d819bfa0 2024-04-27T01:06:13.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.606+0000 7ff6e0775700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6bc005120 con 0x7ff6d819bfa0 2024-04-27T01:06:13.602 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.606+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1547207639 0 0) 0x7ff6d4046390 con 0x7ff6d819bfa0 2024-04-27T01:06:13.605 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.612+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7ff6d4015f60 con 0x7ff6d819bfa0 2024-04-27T01:06:13.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.812+0000 7ff6e0775700 1 -- 172.21.15.162:0/805887509 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ff6bc005410 con 0x7ff6d819bfa0 2024-04-27T01:06:13.807 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.813+0000 7ff6cf7fe700 1 -- 172.21.15.162:0/805887509 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+193 (unknown 2189217731 0 3870630924) 0x7ff6d4015860 con 0x7ff6d819bfa0 2024-04-27T01:06:13.807 INFO:teuthology.orchestra.run.smithi162.stdout:# minimal ceph.conf for ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:06:13.807 INFO:teuthology.orchestra.run.smithi162.stdout:[global] 2024-04-27T01:06:13.807 INFO:teuthology.orchestra.run.smithi162.stdout: fsid = ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:06:13.807 INFO:teuthology.orchestra.run.smithi162.stdout: mon_host = 172.21.15.162:6789/0 172.21.15.181:6789/0 v1:172.21.15.162:6790/0 2024-04-27T01:06:13.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.815+0000 7ff6e0775700 1 -- 172.21.15.162:0/805887509 >> v1:172.21.15.162:6800/2166195007 conn(0x7ff6c403c6c0 legacy=0x7ff6c403eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:13.808 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.815+0000 7ff6e0775700 1 -- 172.21.15.162:0/805887509 >> v1:172.21.15.162:6790/0 conn(0x7ff6d819bfa0 legacy=0x7ff6d81a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:13.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.815+0000 7ff6e0775700 1 -- 172.21.15.162:0/805887509 shutdown_connections 2024-04-27T01:06:13.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.815+0000 7ff6e0775700 1 -- 172.21.15.162:0/805887509 >> 172.21.15.162:0/805887509 conn(0x7ff6d80765c0 msgr2=0x7ff6d8109000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:13.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.815+0000 7ff6e0775700 1 -- 172.21.15.162:0/805887509 shutdown_connections 2024-04-27T01:06:13.809 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:13.815+0000 7ff6e0775700 1 -- 172.21.15.162:0/805887509 wait complete. 2024-04-27T01:06:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[26714]: Updating smithi162:/etc/ceph/ceph.conf 2024-04-27T01:06:14.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[26714]: Updating smithi181:/etc/ceph/ceph.conf 2024-04-27T01:06:14.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[26714]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:06:14.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[26714]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:06:14.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[26714]: Reconfiguring mon.a (unknown last config time)... 2024-04-27T01:06:14.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[26714]: Reconfiguring daemon mon.a on smithi162 2024-04-27T01:06:14.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[26714]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:14.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/805887509' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:14.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[37680]: Updating smithi162:/etc/ceph/ceph.conf 2024-04-27T01:06:14.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[37680]: Updating smithi181:/etc/ceph/ceph.conf 2024-04-27T01:06:14.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[37680]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:06:14.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[37680]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:06:14.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[37680]: Reconfiguring mon.a (unknown last config time)... 2024-04-27T01:06:14.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[37680]: Reconfiguring daemon mon.a on smithi162 2024-04-27T01:06:14.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[37680]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:14.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/805887509' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:14.412 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-27T01:06:14.412 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:06:14.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-04-27T01:06:14.439 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:06:14.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:06:14.505 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:06:14.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.conf 2024-04-27T01:06:14.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:14 smithi181 ceph-mon[30237]: Updating smithi162:/etc/ceph/ceph.conf 2024-04-27T01:06:14.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:14 smithi181 ceph-mon[30237]: Updating smithi181:/etc/ceph/ceph.conf 2024-04-27T01:06:14.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:14 smithi181 ceph-mon[30237]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:06:14.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:14 smithi181 ceph-mon[30237]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:06:14.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:14 smithi181 ceph-mon[30237]: Reconfiguring mon.a (unknown last config time)... 2024-04-27T01:06:14.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:14 smithi181 ceph-mon[30237]: Reconfiguring daemon mon.a on smithi162 2024-04-27T01:06:14.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:14 smithi181 ceph-mon[30237]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:14.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/805887509' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:14.536 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:06:14.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:06:14.604 INFO:tasks.cephadm:Adding mgr.y on smithi162 2024-04-27T01:06:14.604 INFO:tasks.cephadm:Adding mgr.x on smithi181 2024-04-27T01:06:14.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi162=y;smithi181=x' 2024-04-27T01:06:14.879 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:06:16.200 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[26714]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:16.200 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:16.201 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:16.201 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:16.201 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:06:16.201 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:16.201 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[37680]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:16.201 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:16.201 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:16.201 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:16.201 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:06:16.201 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:16 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:16.202 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:16 smithi181 ceph-mon[30237]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:16.202 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:16 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:16.202 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:16 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:16.202 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:16 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:16.202 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:16 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:06:16.202 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:16 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:16.304 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.309+0000 7f9a01ee7700 1 -- 172.21.15.181:0/218429108 >> v1:172.21.15.162:6789/0 conn(0x7f99fc100c20 legacy=0x7f99fc101040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.309+0000 7f9a01ee7700 1 -- 172.21.15.181:0/218429108 shutdown_connections 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.309+0000 7f9a01ee7700 1 -- 172.21.15.181:0/218429108 >> 172.21.15.181:0/218429108 conn(0x7f99fc0fc1c0 msgr2=0x7f99fc0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.309+0000 7f9a01ee7700 1 -- 172.21.15.181:0/218429108 shutdown_connections 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.309+0000 7f9a01ee7700 1 -- 172.21.15.181:0/218429108 wait complete. 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.310+0000 7f9a01ee7700 1 Processor -- start 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.310+0000 7f9a01ee7700 1 -- start start 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.310+0000 7f9a01ee7700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99fc1a1fb0 con 0x7f99fc196e50 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.310+0000 7f9a01ee7700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99fc1a31d0 con 0x7f99fc19e340 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.310+0000 7f9a01ee7700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f99fc1a4410 con 0x7f99fc100c20 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.310+0000 7f99fbfff700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f99fc19e340 0x7f99fc1a0730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:45160/0 (socket says 172.21.15.181:45160) 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.310+0000 7f99fbfff700 1 -- 172.21.15.181:0/510835806 learned_addr learned my addr 172.21.15.181:0/510835806 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1991516275 0 0) 0x7f99fc1a31d0 con 0x7f99fc19e340 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99e8003bf0 con 0x7f99fc19e340 2024-04-27T01:06:16.305 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 875856417 0 0) 0x7f99fc1a1fb0 con 0x7f99fc196e50 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99fc1a31d0 con 0x7f99fc196e50 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 631490119 0 0) 0x7f99fc1a4410 con 0x7f99fc100c20 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99fc1a1fb0 con 0x7f99fc100c20 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2856557412 0 0) 0x7f99e8003bf0 con 0x7f99fc19e340 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99fc1a4410 con 0x7f99fc19e340 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3771755351 0 0) 0x7f99fc1a31d0 con 0x7f99fc196e50 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99e8003bf0 con 0x7f99fc196e50 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f99f4001b70 con 0x7f99fc19e340 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4028834472 0 0) 0x7f99fc1a1fb0 con 0x7f99fc100c20 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f99fc1a31d0 con 0x7f99fc100c20 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f99e4001790 con 0x7f99fc196e50 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f99ec003910 con 0x7f99fc100c20 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1535399214 0 0) 0x7f99fc1a4410 con 0x7f99fc19e340 2024-04-27T01:06:16.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 >> v1:172.21.15.162:6790/0 conn(0x7f99fc100c20 legacy=0x7f99fc196740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:16.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 >> v1:172.21.15.162:6789/0 conn(0x7f99fc196e50 legacy=0x7f99fc19cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:16.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99fc1a5650 con 0x7f99fc19e340 2024-04-27T01:06:16.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f9a01ee7700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f99fc1a21e0 con 0x7f99fc19e340 2024-04-27T01:06:16.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f9a01ee7700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f99fc1a2720 con 0x7f99fc19e340 2024-04-27T01:06:16.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f99f40051c0 con 0x7f99fc19e340 2024-04-27T01:06:16.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.311+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f99f4006050 con 0x7f99fc19e340 2024-04-27T01:06:16.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.312+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7f99f40117c0 con 0x7f99fc19e340 2024-04-27T01:06:16.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.312+0000 7f9a01ee7700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99dc005150 con 0x7f99fc19e340 2024-04-27T01:06:16.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.313+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1547207639 0 0) 0x7f99f4047c10 con 0x7f99fc19e340 2024-04-27T01:06:16.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.317+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f99f4017660 con 0x7f99fc19e340 2024-04-27T01:06:16.521 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.529+0000 7f9a01ee7700 1 -- 172.21.15.181:0/510835806 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi162=y;smithi181=x", "target": ["mon-mgr", ""]}) -- 0x7f99dc000bc0 con 0x7f99e803a4a0 2024-04-27T01:06:16.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.535+0000 7f99f97fa700 1 -- 172.21.15.181:0/510835806 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (unknown 0 0 325935098) 0x7f99dc000bc0 con 0x7f99e803a4a0 2024-04-27T01:06:16.527 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mgr update... 2024-04-27T01:06:16.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.537+0000 7f9a01ee7700 1 -- 172.21.15.181:0/510835806 >> v1:172.21.15.162:6800/2166195007 conn(0x7f99e803a4a0 legacy=0x7f99e803c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:16.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.537+0000 7f9a01ee7700 1 -- 172.21.15.181:0/510835806 >> v1:172.21.15.181:6789/0 conn(0x7f99fc19e340 legacy=0x7f99fc1a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:16.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.537+0000 7f9a01ee7700 1 -- 172.21.15.181:0/510835806 shutdown_connections 2024-04-27T01:06:16.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.537+0000 7f9a01ee7700 1 -- 172.21.15.181:0/510835806 >> 172.21.15.181:0/510835806 conn(0x7f99fc0fc1c0 msgr2=0x7f99fc103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:16.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.537+0000 7f9a01ee7700 1 -- 172.21.15.181:0/510835806 shutdown_connections 2024-04-27T01:06:16.530 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:06:16.537+0000 7f9a01ee7700 1 -- 172.21.15.181:0/510835806 wait complete. 2024-04-27T01:06:17.132 DEBUG:teuthology.orchestra.run.smithi181:mgr.x> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mgr.x.service 2024-04-27T01:06:17.135 INFO:tasks.cephadm:Deploying OSDs... 2024-04-27T01:06:17.135 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:06:17.135 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T01:06:17.154 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T01:06:17.154 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-04-27T01:06:17.212 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T01:06:17.212 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:06:17.212 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-27T01:06:17.212 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:06:17.212 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:06:17.212 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-04-27 01:06:05.004951064 +0000 2024-04-27T01:06:17.212 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-04-27 00:59:21.341812511 +0000 2024-04-27T01:06:17.212 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-04-27 00:59:21.341812511 +0000 2024-04-27T01:06:17.212 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-04-27 00:59:21.341812511 +0000 2024-04-27T01:06:17.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T01:06:17.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:17 smithi181 ceph-mon[30237]: Reconfiguring mon.c (monmap changed)... 2024-04-27T01:06:17.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:17 smithi181 ceph-mon[30237]: Reconfiguring daemon mon.c on smithi162 2024-04-27T01:06:17.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:17 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:17.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:17 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:17.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:17 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:17.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:17 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:17.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:17 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:06:17.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:17 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:17.278 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-04-27T01:06:17.278 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-04-27T01:06:17.278 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.00025999 s, 2.0 MB/s 2024-04-27T01:06:17.279 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T01:06:17.337 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-04-27T01:06:17.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[26714]: Reconfiguring mon.c (monmap changed)... 2024-04-27T01:06:17.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[26714]: Reconfiguring daemon mon.c on smithi162 2024-04-27T01:06:17.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:17.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:17.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:17.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:17.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:06:17.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:17.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[37680]: Reconfiguring mon.c (monmap changed)... 2024-04-27T01:06:17.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[37680]: Reconfiguring daemon mon.c on smithi162 2024-04-27T01:06:17.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:17.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:17.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:17.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-27T01:06:17.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-27T01:06:17.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:17 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:17.358 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T01:06:17.358 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:06:17.359 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-27T01:06:17.359 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:06:17.359 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:06:17.359 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-04-27 01:06:05.005951046 +0000 2024-04-27T01:06:17.359 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-04-27 00:59:21.682805690 +0000 2024-04-27T01:06:17.359 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-04-27 00:59:21.682805690 +0000 2024-04-27T01:06:17.359 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-04-27 00:59:21.682805690 +0000 2024-04-27T01:06:17.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T01:06:17.423 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-04-27T01:06:17.423 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-04-27T01:06:17.423 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000211971 s, 2.4 MB/s 2024-04-27T01:06:17.425 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T01:06:17.481 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-04-27T01:06:17.537 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T01:06:17.537 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:06:17.537 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-27T01:06:17.537 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:06:17.538 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:06:17.538 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-04-27 01:06:05.005951046 +0000 2024-04-27T01:06:17.538 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-04-27 00:59:22.027798788 +0000 2024-04-27T01:06:17.538 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-04-27 00:59:22.027798788 +0000 2024-04-27T01:06:17.538 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-04-27 00:59:22.027798788 +0000 2024-04-27T01:06:17.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T01:06:17.601 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-04-27T01:06:17.601 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-04-27T01:06:17.601 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000188221 s, 2.7 MB/s 2024-04-27T01:06:17.602 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T01:06:17.658 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-04-27T01:06:17.713 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T01:06:17.713 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:06:17.713 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-27T01:06:17.713 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:06:17.713 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:06:17.713 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-04-27 01:06:05.005951046 +0000 2024-04-27T01:06:17.713 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-04-27 00:59:22.363792067 +0000 2024-04-27T01:06:17.713 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-04-27 00:59:22.363792067 +0000 2024-04-27T01:06:17.713 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-04-27 00:59:22.363792067 +0000 2024-04-27T01:06:17.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T01:06:17.775 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-04-27T01:06:17.775 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-04-27T01:06:17.775 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000198173 s, 2.6 MB/s 2024-04-27T01:06:17.776 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T01:06:17.831 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:06:17.831 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T01:06:17.847 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T01:06:17.847 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-04-27T01:06:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T01:06:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:06:17.903 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-27T01:06:17.903 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:06:17.903 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:06:17.903 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-27 01:05:56.065536484 +0000 2024-04-27T01:06:17.903 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-27 00:59:18.412942973 +0000 2024-04-27T01:06:17.904 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-27 00:59:18.412942973 +0000 2024-04-27T01:06:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-04-27 00:59:18.412942973 +0000 2024-04-27T01:06:17.904 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T01:06:17.966 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-27T01:06:17.967 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-27T01:06:17.967 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000192016 s, 2.7 MB/s 2024-04-27T01:06:17.968 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T01:06:18.023 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-04-27T01:06:18.077 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T01:06:18.077 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:06:18.077 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-27T01:06:18.078 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:06:18.078 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:06:18.078 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-27 01:05:56.065536484 +0000 2024-04-27T01:06:18.078 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-27 00:59:18.760935469 +0000 2024-04-27T01:06:18.078 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-27 00:59:18.760935469 +0000 2024-04-27T01:06:18.078 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-04-27 00:59:18.760935469 +0000 2024-04-27T01:06:18.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T01:06:18.140 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-27T01:06:18.140 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-27T01:06:18.140 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000196168 s, 2.6 MB/s 2024-04-27T01:06:18.141 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T01:06:18.198 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-04-27T01:06:18.253 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T01:06:18.253 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:06:18.253 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-27T01:06:18.254 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:06:18.254 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:06:18.254 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-27 01:05:56.065536484 +0000 2024-04-27T01:06:18.254 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-27 00:59:19.110927921 +0000 2024-04-27T01:06:18.254 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-27 00:59:19.110927921 +0000 2024-04-27T01:06:18.254 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-04-27 00:59:19.110927921 +0000 2024-04-27T01:06:18.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T01:06:18.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:18 smithi181 ceph-mon[30237]: from='client.24103 v1:172.21.15.181:0/510835806' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi162=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:18.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:18 smithi181 ceph-mon[30237]: Saving service mgr spec with placement smithi162=y;smithi181=x;count:2 2024-04-27T01:06:18.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:18 smithi181 ceph-mon[30237]: Reconfiguring mon.b (monmap changed)... 2024-04-27T01:06:18.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:18 smithi181 ceph-mon[30237]: Reconfiguring daemon mon.b on smithi181 2024-04-27T01:06:18.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:18 smithi181 ceph-mon[30237]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:18.284 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-27T01:06:18.285 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-27T01:06:18.285 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000173261 s, 3.0 MB/s 2024-04-27T01:06:18.286 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T01:06:18.341 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-04-27T01:06:18.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:18 smithi162 ceph-mon[26714]: from='client.24103 v1:172.21.15.181:0/510835806' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi162=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:18.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:18 smithi162 ceph-mon[26714]: Saving service mgr spec with placement smithi162=y;smithi181=x;count:2 2024-04-27T01:06:18.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:18 smithi162 ceph-mon[26714]: Reconfiguring mon.b (monmap changed)... 2024-04-27T01:06:18.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:18 smithi162 ceph-mon[26714]: Reconfiguring daemon mon.b on smithi181 2024-04-27T01:06:18.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:18 smithi162 ceph-mon[26714]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:18.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:18 smithi162 ceph-mon[37680]: from='client.24103 v1:172.21.15.181:0/510835806' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi162=y;smithi181=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:18.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:18 smithi162 ceph-mon[37680]: Saving service mgr spec with placement smithi162=y;smithi181=x;count:2 2024-04-27T01:06:18.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:18 smithi162 ceph-mon[37680]: Reconfiguring mon.b (monmap changed)... 2024-04-27T01:06:18.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:18 smithi162 ceph-mon[37680]: Reconfiguring daemon mon.b on smithi181 2024-04-27T01:06:18.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:18 smithi162 ceph-mon[37680]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:18.396 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T01:06:18.396 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T01:06:18.396 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-27T01:06:18.396 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T01:06:18.396 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-04-27T01:06:18.396 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-27 01:05:56.066536465 +0000 2024-04-27T01:06:18.396 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-27 00:59:19.452920546 +0000 2024-04-27T01:06:18.396 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-27 00:59:19.452920546 +0000 2024-04-27T01:06:18.397 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-04-27 00:59:19.452920546 +0000 2024-04-27T01:06:18.397 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T01:06:18.458 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-27T01:06:18.458 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-27T01:06:18.458 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000179526 s, 2.9 MB/s 2024-04-27T01:06:18.460 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T01:06:18.515 INFO:tasks.cephadm:Deploying osd.0 on smithi162 with /dev/vg_nvme/lv_4... 2024-04-27T01:06:18.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-27T01:06:18.748 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:06:19.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:19.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:19.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:19.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:19.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:19.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:19.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:06:19.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-27T01:06:19.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:06:19.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:19.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: Deploying daemon mgr.x on smithi181 2024-04-27T01:06:19.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:19 smithi181 ceph-mon[30237]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:19.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: Deploying daemon mgr.x on smithi181 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[26714]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:19.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: Deploying daemon mgr.x on smithi181 2024-04-27T01:06:19.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:19 smithi162 ceph-mon[37680]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:20.445 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:20 smithi181 systemd[1]: Starting Ceph mgr.x for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:06:20.619 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-27T01:06:20.619 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-27T01:06:20.619 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-04-27T01:06:20.620 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-04-27T01:06:20.620 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113816 s, 921 MB/s 2024-04-27T01:06:20.620 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-04-27T01:06:20.697 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:20 smithi181 podman[32075]: 2024-04-27 01:06:20.452820897 +0000 UTC m=+0.108562410 container create ef824765aa95b8b8c2760e90d29a0109f6984f73ad31ccb111904b02ec440f90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-27T01:06:20.697 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:20 smithi181 podman[32075]: 2024-04-27 01:06:20.362169605 +0000 UTC m=+0.017911076 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:06:20.697 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:20 smithi181 podman[32075]: 2024-04-27 01:06:20.630063547 +0000 UTC m=+0.285805018 container init ef824765aa95b8b8c2760e90d29a0109f6984f73ad31ccb111904b02ec440f90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-27T01:06:20.697 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:20 smithi181 podman[32075]: 2024-04-27 01:06:20.632585464 +0000 UTC m=+0.288326944 container start ef824765aa95b8b8c2760e90d29a0109f6984f73ad31ccb111904b02ec440f90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-27T01:06:20.697 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:20 smithi181 bash[32075]: ef824765aa95b8b8c2760e90d29a0109f6984f73ad31ccb111904b02ec440f90 2024-04-27T01:06:20.976 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:20 smithi181 systemd[1]: Started Ceph mgr.x for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:06:20.976 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:20 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:20.794+0000 7f4b686cd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-27T01:06:20.976 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:20 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:20.984+0000 7f4b686cd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:06:21.263 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:21 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:21.094+0000 7f4b686cd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-27T01:06:21.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-04-27T01:06:21.564 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:06:21.763 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:21 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:21.428+0000 7f4b686cd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-27T01:06:21.764 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:21 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:21.524+0000 7f4b686cd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-27T01:06:21.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:21.817 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:21.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:21.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:21.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:21.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[26714]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:21.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:21.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:21.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:21.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:21.818 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:21.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:21.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[37680]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:21.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:22.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:22.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:22.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:22.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:22.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:22.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:21 smithi181 ceph-mon[30237]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:22.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:23.013 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:22.745+0000 7f4b686cd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-27T01:06:23.014 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:22.914+0000 7f4b686cd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-27T01:06:23.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.088+0000 7f904663a700 1 -- 172.21.15.162:0/2358727250 >> v1:172.21.15.162:6790/0 conn(0x7f9040102b60 legacy=0x7f9040104f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:23.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.088+0000 7f904663a700 1 -- 172.21.15.162:0/2358727250 shutdown_connections 2024-04-27T01:06:23.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.088+0000 7f904663a700 1 -- 172.21.15.162:0/2358727250 >> 172.21.15.162:0/2358727250 conn(0x7f90400fc550 msgr2=0x7f90400fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:23.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.088+0000 7f904663a700 1 -- 172.21.15.162:0/2358727250 shutdown_connections 2024-04-27T01:06:23.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.088+0000 7f904663a700 1 -- 172.21.15.162:0/2358727250 wait complete. 2024-04-27T01:06:23.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f904663a700 1 Processor -- start 2024-04-27T01:06:23.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f904663a700 1 -- start start 2024-04-27T01:06:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f904663a700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9040197f30 con 0x7f9040106670 2024-04-27T01:06:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f904663a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f90401a3720 con 0x7f9040102b60 2024-04-27T01:06:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f904663a700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f90401a4900 con 0x7f904010a1b0 2024-04-27T01:06:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f9044bd7700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f904010a1b0 0x7f90401a0ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:35666/0 (socket says 172.21.15.162:35666) 2024-04-27T01:06:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f9044bd7700 1 -- 172.21.15.162:0/1717545536 learned_addr learned my addr 172.21.15.162:0/1717545536 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:06:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 488345163 0 0) 0x7f90401a3720 con 0x7f9040102b60 2024-04-27T01:06:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90280035b0 con 0x7f9040102b60 2024-04-27T01:06:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2554432366 0 0) 0x7f90401a4900 con 0x7f904010a1b0 2024-04-27T01:06:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90401a3720 con 0x7f904010a1b0 2024-04-27T01:06:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2674899031 0 0) 0x7f9040197f30 con 0x7f9040106670 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.089+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90401a4900 con 0x7f9040106670 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1169368514 0 0) 0x7f90280035b0 con 0x7f9040102b60 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9040197f30 con 0x7f9040102b60 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f9038001bc0 con 0x7f9040102b60 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1647965099 0 0) 0x7f90401a3720 con 0x7f904010a1b0 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90280035b0 con 0x7f904010a1b0 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 289080166 0 0) 0x7f90401a4900 con 0x7f9040106670 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90401a3720 con 0x7f9040106670 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f9030001200 con 0x7f904010a1b0 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f90340017f0 con 0x7f9040106670 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 30703612 0 0) 0x7f9040197f30 con 0x7f9040102b60 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 >> v1:172.21.15.162:6790/0 conn(0x7f904010a1b0 legacy=0x7f90401a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 >> v1:172.21.15.162:6789/0 conn(0x7f9040106670 legacy=0x7f904019d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:23.084 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90401a5b20 con 0x7f9040102b60 2024-04-27T01:06:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f904663a700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f90401a4b30 con 0x7f9040102b60 2024-04-27T01:06:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f904663a700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f90401a5070 con 0x7f9040102b60 2024-04-27T01:06:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f90380049d0 con 0x7f9040102b60 2024-04-27T01:06:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.090+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f90380050f0 con 0x7f9040102b60 2024-04-27T01:06:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.091+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 13) ==== 46309+0+0 (unknown 1841890923 0 0) 0x7f9038010780 con 0x7f9040102b60 2024-04-27T01:06:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.091+0000 7f904663a700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f902c005150 con 0x7f9040102b60 2024-04-27T01:06:23.085 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.091+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 1069+0+0 (unknown 1547207639 0 0) 0x7f9038001f10 con 0x7f9040102b60 2024-04-27T01:06:23.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.096+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f90380166d0 con 0x7f9040102b60 2024-04-27T01:06:23.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:23.310+0000 7f904663a700 1 -- 172.21.15.162:0/1717545536 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f902c000bc0 con 0x7f902803a420 2024-04-27T01:06:23.364 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:23.085+0000 7f4b686cd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-27T01:06:23.364 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:23.372+0000 7f4b686cd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-27T01:06:23.364 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:23.364 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:23.364 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.364 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:23.364 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:23.364 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: Reconfiguring mgr.y (unknown last config time)... 2024-04-27T01:06:23.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:06:23.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:06:23.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: Reconfiguring daemon mgr.y on smithi162 2024-04-27T01:06:23.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-mon[30237]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: Reconfiguring mgr.y (unknown last config time)... 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: Reconfiguring daemon mgr.y on smithi162 2024-04-27T01:06:23.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[37680]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: Reconfiguring mgr.y (unknown last config time)... 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: Reconfiguring daemon mgr.y on smithi162 2024-04-27T01:06:23.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:23 smithi162 ceph-mon[26714]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:23.658 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:23.464+0000 7f4b686cd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-27T01:06:23.659 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:23.666+0000 7f4b686cd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-27T01:06:24.013 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:23.856+0000 7f4b686cd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:06:24.502 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[26714]: from='client.24115 v1:172.21.15.162:0/1717545536' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:24.502 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:06:24.502 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:06:24.502 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:24.502 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:24.502 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:24.502 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[37680]: from='client.24115 v1:172.21.15.162:0/1717545536' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:24.502 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:06:24.502 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:06:24.503 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:24.503 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:24.503 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:24 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:24.513 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:24.457+0000 7f4b686cd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-27T01:06:24.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:24 smithi181 ceph-mon[30237]: from='client.24115 v1:172.21.15.162:0/1717545536' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:24.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:24 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:06:24.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:24 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:06:24.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:24 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:24.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:24 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:24.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:24 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:25.168 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:24.876+0000 7f4b686cd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-27T01:06:25.168 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:24.972+0000 7f4b686cd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-27T01:06:25.168 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:25 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:25.175+0000 7f4b686cd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-27T01:06:25.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:25.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:25.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:25.492 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:25.492 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:25 smithi162 ceph-mon[26714]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:25.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:25.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:25.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:25.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:25.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:25 smithi162 ceph-mon[37680]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:25.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:25.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:25.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:25.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:25.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:25 smithi181 ceph-mon[30237]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:26.013 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:25 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:25.700+0000 7f4b686cd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-27T01:06:26.013 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:25 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:25.791+0000 7f4b686cd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-27T01:06:26.367 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:26.035+0000 7f4b686cd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-27T01:06:26.368 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:26.166+0000 7f4b686cd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-27T01:06:26.368 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:26.260+0000 7f4b686cd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-27T01:06:26.368 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:26.374+0000 7f4b686cd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-27T01:06:26.763 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:26.465+0000 7f4b686cd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-27T01:06:27.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2757106204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f1e299d-4d29-4333-b632-249ff9f12d5d"}]: dispatch 2024-04-27T01:06:27.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-mon[30237]: from='client.34118 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f1e299d-4d29-4333-b632-249ff9f12d5d"}]: dispatch 2024-04-27T01:06:27.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-mon[30237]: from='client.34118 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f1e299d-4d29-4333-b632-249ff9f12d5d"}]': finished 2024-04-27T01:06:27.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-mon[30237]: osdmap e5: 1 total, 0 up, 1 in 2024-04-27T01:06:27.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:27.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/747277331' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:06:27.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2757106204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f1e299d-4d29-4333-b632-249ff9f12d5d"}]: dispatch 2024-04-27T01:06:27.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[37680]: from='client.34118 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f1e299d-4d29-4333-b632-249ff9f12d5d"}]: dispatch 2024-04-27T01:06:27.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[37680]: from='client.34118 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f1e299d-4d29-4333-b632-249ff9f12d5d"}]': finished 2024-04-27T01:06:27.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[37680]: osdmap e5: 1 total, 0 up, 1 in 2024-04-27T01:06:27.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:27.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/747277331' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:06:27.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2757106204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f1e299d-4d29-4333-b632-249ff9f12d5d"}]: dispatch 2024-04-27T01:06:27.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[26714]: from='client.34118 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f1e299d-4d29-4333-b632-249ff9f12d5d"}]: dispatch 2024-04-27T01:06:27.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[26714]: from='client.34118 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f1e299d-4d29-4333-b632-249ff9f12d5d"}]': finished 2024-04-27T01:06:27.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[26714]: osdmap e5: 1 total, 0 up, 1 in 2024-04-27T01:06:27.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:27.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/747277331' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:06:27.763 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:27.276+0000 7f4b686cd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-27T01:06:27.763 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:06:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:06:27.490+0000 7f4b686cd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-27T01:06:27.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:27.973+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mgrmap(e 14) ==== 92026+0+0 (unknown 428087007 0 0) 0x7f90380461c0 con 0x7f9040102b60 2024-04-27T01:06:28.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:27 smithi181 ceph-mon[30237]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:28.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:27 smithi181 ceph-mon[30237]: Standby manager daemon x started 2024-04-27T01:06:28.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:27 smithi181 ceph-mon[30237]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-27T01:06:28.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:27 smithi181 ceph-mon[30237]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:06:28.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:27 smithi181 ceph-mon[30237]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-27T01:06:28.264 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:27 smithi181 ceph-mon[30237]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:06:28.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[26714]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:28.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[26714]: Standby manager daemon x started 2024-04-27T01:06:28.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[26714]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-27T01:06:28.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[26714]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:06:28.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[26714]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-27T01:06:28.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[26714]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:06:28.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[37680]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:28.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[37680]: Standby manager daemon x started 2024-04-27T01:06:28.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[37680]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-27T01:06:28.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[37680]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:06:28.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[37680]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-27T01:06:28.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:27 smithi162 ceph-mon[37680]: from='mgr.? v1:172.21.15.181:0/2666824829' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:06:29.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:28 smithi181 ceph-mon[30237]: mgrmap e14: y(active, since 67s), standbys: x 2024-04-27T01:06:29.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:28 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-27T01:06:29.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:28 smithi162 ceph-mon[37680]: mgrmap e14: y(active, since 67s), standbys: x 2024-04-27T01:06:29.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:28 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-27T01:06:29.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:28 smithi162 ceph-mon[26714]: mgrmap e14: y(active, since 67s), standbys: x 2024-04-27T01:06:29.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:28 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-27T01:06:30.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:29 smithi181 ceph-mon[30237]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:30.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:29 smithi162 ceph-mon[37680]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:30.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:29 smithi162 ceph-mon[26714]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:32.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:31 smithi181 ceph-mon[30237]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:32.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:31 smithi162 ceph-mon[26714]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:32.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:31 smithi162 ceph-mon[37680]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:33.240 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:32 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T01:06:33.240 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:32 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:33.240 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:32 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T01:06:33.240 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:32 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:33.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:32 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-27T01:06:33.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:32 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:34.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:33 smithi181 ceph-mon[30237]: Deploying daemon osd.0 on smithi162 2024-04-27T01:06:34.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:33 smithi181 ceph-mon[30237]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:34.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:33 smithi162 ceph-mon[26714]: Deploying daemon osd.0 on smithi162 2024-04-27T01:06:34.281 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:33 smithi162 ceph-mon[26714]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:34.281 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:33 smithi162 ceph-mon[37680]: Deploying daemon osd.0 on smithi162 2024-04-27T01:06:34.281 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:33 smithi162 ceph-mon[37680]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:36.162 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:35 smithi162 ceph-mon[26714]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:36.163 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:35 smithi162 ceph-mon[37680]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:36.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:35 smithi181 ceph-mon[30237]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:38.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:37 smithi181 ceph-mon[30237]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:38.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:38.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:38.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:38.289 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:37 smithi162 ceph-mon[37680]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:38.290 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:38.290 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:38.290 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:38.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:37 smithi162 ceph-mon[26714]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:38.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:38.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:38.290 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:40.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:39 smithi181 ceph-mon[30237]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:40.271 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:39 smithi162 ceph-mon[26714]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:40.271 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:39 smithi162 ceph-mon[37680]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:41.604 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 0 on host 'smithi162' 2024-04-27T01:06:41.604 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:41.611+0000 7f903d7fa700 1 -- 172.21.15.162:0/1717545536 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 2947205282) 0x7f902c000bc0 con 0x7f902803a420 2024-04-27T01:06:41.606 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:41.614+0000 7f904663a700 1 -- 172.21.15.162:0/1717545536 >> v1:172.21.15.162:6800/2166195007 conn(0x7f902803a420 legacy=0x7f902803c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:41.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:41.614+0000 7f904663a700 1 -- 172.21.15.162:0/1717545536 >> v1:172.21.15.181:6789/0 conn(0x7f9040102b60 legacy=0x7f90401973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:41.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:41.614+0000 7f904663a700 1 -- 172.21.15.162:0/1717545536 shutdown_connections 2024-04-27T01:06:41.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:41.614+0000 7f904663a700 1 -- 172.21.15.162:0/1717545536 >> 172.21.15.162:0/1717545536 conn(0x7f90400fc550 msgr2=0x7f904010c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:41.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:41.614+0000 7f904663a700 1 -- 172.21.15.162:0/1717545536 shutdown_connections 2024-04-27T01:06:41.607 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:41.614+0000 7f904663a700 1 -- 172.21.15.162:0/1717545536 wait complete. 2024-04-27T01:06:41.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:41.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:41.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:41.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:41.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:41.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[26714]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:41.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[26714]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T01:06:41.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:41.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:41.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:41.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:41.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:41.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:41.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:41.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:41.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[37680]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:41.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[37680]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T01:06:41.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:41.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:41.843 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:41 smithi181 ceph-mon[30237]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:41 smithi181 ceph-mon[30237]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-27T01:06:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:06:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:42.256 DEBUG:teuthology.orchestra.run.smithi162:osd.0> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.0.service 2024-04-27T01:06:42.258 INFO:tasks.cephadm:Deploying osd.1 on smithi162 with /dev/vg_nvme/lv_3... 2024-04-27T01:06:42.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-27T01:06:42.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:42 smithi162 ceph-mon[26714]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T01:06:42.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:42 smithi162 ceph-mon[26714]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T01:06:42.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:42 smithi162 ceph-mon[26714]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:06:42.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:42 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:43.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:42 smithi181 ceph-mon[30237]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T01:06:43.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:42 smithi181 ceph-mon[30237]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T01:06:43.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:42 smithi181 ceph-mon[30237]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:06:43.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:42 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:43.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:42 smithi162 ceph-mon[37680]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-27T01:06:43.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:42 smithi162 ceph-mon[37680]: osdmap e6: 1 total, 0 up, 1 in 2024-04-27T01:06:43.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:42 smithi162 ceph-mon[37680]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:06:43.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:42 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:43.422 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0[41669]: 2024-04-27T01:06:43.155+0000 7fda6bdb7700 -1 osd.0 0 waiting for initial osdmap 2024-04-27T01:06:43.422 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0[41669]: 2024-04-27T01:06:43.158+0000 7fda633a1700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:06:43.529 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:06:44.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:43 smithi181 ceph-mon[30237]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:06:44.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:43 smithi181 ceph-mon[30237]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T01:06:44.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:43 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:44.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:43 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:44.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:43 smithi181 ceph-mon[30237]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:44.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:43 smithi181 ceph-mon[30237]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' 2024-04-27T01:06:44.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:43 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:44.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[26714]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:06:44.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[26714]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T01:06:44.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:44.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:44.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[26714]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:44.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[26714]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' 2024-04-27T01:06:44.035 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:44.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[37680]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:06:44.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[37680]: osdmap e7: 1 total, 0 up, 1 in 2024-04-27T01:06:44.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:44.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:44.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[37680]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-27T01:06:44.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[37680]: from='osd.0 v1:172.21.15.162:6801/2041814805' entity='osd.0' 2024-04-27T01:06:44.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:43 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:45.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:44 smithi162 ceph-mon[26714]: purged_snaps scrub starts 2024-04-27T01:06:45.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:44 smithi162 ceph-mon[26714]: purged_snaps scrub ok 2024-04-27T01:06:45.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:44 smithi162 ceph-mon[26714]: osd.0 v1:172.21.15.162:6801/2041814805 boot 2024-04-27T01:06:45.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:44 smithi162 ceph-mon[26714]: osdmap e8: 1 total, 1 up, 1 in 2024-04-27T01:06:45.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:44 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:45.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:44 smithi162 ceph-mon[37680]: purged_snaps scrub starts 2024-04-27T01:06:45.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:44 smithi162 ceph-mon[37680]: purged_snaps scrub ok 2024-04-27T01:06:45.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:44 smithi162 ceph-mon[37680]: osd.0 v1:172.21.15.162:6801/2041814805 boot 2024-04-27T01:06:45.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:44 smithi162 ceph-mon[37680]: osdmap e8: 1 total, 1 up, 1 in 2024-04-27T01:06:45.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:44 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:45.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:44 smithi181 ceph-mon[30237]: purged_snaps scrub starts 2024-04-27T01:06:45.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:44 smithi181 ceph-mon[30237]: purged_snaps scrub ok 2024-04-27T01:06:45.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:44 smithi181 ceph-mon[30237]: osd.0 v1:172.21.15.162:6801/2041814805 boot 2024-04-27T01:06:45.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:44 smithi181 ceph-mon[30237]: osdmap e8: 1 total, 1 up, 1 in 2024-04-27T01:06:45.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:44 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: Detected new or changed devices on smithi162 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi162 to 16064M 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.247 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[26714]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: Detected new or changed devices on smithi162 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi162 to 16064M 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.248 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:46 smithi162 ceph-mon[37680]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T01:06:46.249 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-27T01:06:46.249 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-27T01:06:46.249 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-04-27T01:06:46.249 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-04-27T01:06:46.249 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01132 s, 926 MB/s 2024-04-27T01:06:46.249 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-04-27T01:06:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: Detected new or changed devices on smithi162 2024-04-27T01:06:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:06:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi162 to 16064M 2024-04-27T01:06:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:06:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:06:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:46 smithi181 ceph-mon[30237]: osdmap e9: 1 total, 1 up, 1 in 2024-04-27T01:06:46.836 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-04-27T01:06:47.072 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:06:48.235 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:48 smithi162 ceph-mon[37680]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:48.344 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:48 smithi162 ceph-mon[26714]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:48.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.450+0000 7f117f326700 1 -- 172.21.15.162:0/4060359261 >> v1:172.21.15.162:6789/0 conn(0x7f1178102b20 legacy=0x7f1178104f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:48.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.450+0000 7f117f326700 1 -- 172.21.15.162:0/4060359261 shutdown_connections 2024-04-27T01:06:48.444 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.450+0000 7f117f326700 1 -- 172.21.15.162:0/4060359261 >> 172.21.15.162:0/4060359261 conn(0x7f11780fc550 msgr2=0x7f11780fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.450+0000 7f117f326700 1 -- 172.21.15.162:0/4060359261 shutdown_connections 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.450+0000 7f117f326700 1 -- 172.21.15.162:0/4060359261 wait complete. 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f117f326700 1 Processor -- start 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f117f326700 1 -- start start 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f117f326700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1178197f50 con 0x7f1178102b20 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f117f326700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f11781a3740 con 0x7f1178106630 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f117f326700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f11781a4920 con 0x7f117810a1d0 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f117d0c2700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f1178102b20 0x7f1178197400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:59304/0 (socket says 172.21.15.162:59304) 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f117d8c3700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f117810a1d0 0x7f11781a1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:55696/0 (socket says 172.21.15.162:55696) 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f117d0c2700 1 -- 172.21.15.162:0/1220260657 learned_addr learned my addr 172.21.15.162:0/1220260657 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4252167884 0 0) 0x7f11781a4920 con 0x7f117810a1d0 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11640035b0 con 0x7f117810a1d0 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2831192064 0 0) 0x7f1178197f50 con 0x7f1178102b20 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11781a4920 con 0x7f1178102b20 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3543024141 0 0) 0x7f11781a3740 con 0x7f1178106630 2024-04-27T01:06:48.445 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.451+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1178197f50 con 0x7f1178106630 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3423068336 0 0) 0x7f11640035b0 con 0x7f117810a1d0 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11781a3740 con 0x7f117810a1d0 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1170001c30 con 0x7f117810a1d0 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3899674035 0 0) 0x7f1178197f50 con 0x7f1178106630 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11640035b0 con 0x7f1178106630 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1174001870 con 0x7f1178106630 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 547399911 0 0) 0x7f11781a4920 con 0x7f1178102b20 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1178197f50 con 0x7f1178102b20 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1168001440 con 0x7f1178102b20 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3889170843 0 0) 0x7f11781a3740 con 0x7f117810a1d0 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 >> v1:172.21.15.181:6789/0 conn(0x7f1178106630 legacy=0x7f117819d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 >> v1:172.21.15.162:6789/0 conn(0x7f1178102b20 legacy=0x7f1178197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11781a5b40 con 0x7f117810a1d0 2024-04-27T01:06:48.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f117f326700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f11781a4b50 con 0x7f117810a1d0 2024-04-27T01:06:48.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f117f326700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f11781a5090 con 0x7f117810a1d0 2024-04-27T01:06:48.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f1170001fc0 con 0x7f117810a1d0 2024-04-27T01:06:48.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.452+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1170006090 con 0x7f117810a1d0 2024-04-27T01:06:48.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.453+0000 7f117f326700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115c005150 con 0x7f117810a1d0 2024-04-27T01:06:48.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.453+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 14) ==== 92026+0+0 (unknown 428087007 0 0) 0x7f1170003cd0 con 0x7f117810a1d0 2024-04-27T01:06:48.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.454+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(9..9 src has 1..9) ==== 1541+0+0 (unknown 2155867406 0 0) 0x7f11700883d0 con 0x7f117810a1d0 2024-04-27T01:06:48.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.459+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f1170056e80 con 0x7f117810a1d0 2024-04-27T01:06:48.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:48 smithi181 ceph-mon[30237]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:48.656 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:06:48.664+0000 7f117f326700 1 -- 172.21.15.162:0/1220260657 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f115c000bc0 con 0x7f116406ef10 2024-04-27T01:06:49.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:49 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:06:49.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:49 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:06:49.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:49 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:49.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:49 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:06:49.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:49 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:06:49.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:49 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:49.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:49 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:06:49.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:49 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:06:49.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:49 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:50.286 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:50 smithi162 ceph-mon[26714]: from='client.34139 v1:172.21.15.162:0/1220260657' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:50.286 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:50 smithi162 ceph-mon[26714]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:50.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:50 smithi162 ceph-mon[37680]: from='client.34139 v1:172.21.15.162:0/1220260657' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:50.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:50 smithi162 ceph-mon[37680]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:50.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:50 smithi181 ceph-mon[30237]: from='client.34139 v1:172.21.15.162:0/1220260657' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:06:50.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:50 smithi181 ceph-mon[30237]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[26714]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1833879508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9013bcef-adce-49f2-952c-f3f208bf4e96"}]: dispatch 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[26714]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9013bcef-adce-49f2-952c-f3f208bf4e96"}]: dispatch 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[26714]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9013bcef-adce-49f2-952c-f3f208bf4e96"}]': finished 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[26714]: osdmap e10: 2 total, 1 up, 2 in 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4286076989' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[37680]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1833879508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9013bcef-adce-49f2-952c-f3f208bf4e96"}]: dispatch 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[37680]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9013bcef-adce-49f2-952c-f3f208bf4e96"}]: dispatch 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[37680]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9013bcef-adce-49f2-952c-f3f208bf4e96"}]': finished 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[37680]: osdmap e10: 2 total, 1 up, 2 in 2024-04-27T01:06:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:06:52.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4286076989' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:06:52.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:52 smithi181 ceph-mon[30237]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:52.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1833879508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9013bcef-adce-49f2-952c-f3f208bf4e96"}]: dispatch 2024-04-27T01:06:52.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:52 smithi181 ceph-mon[30237]: from='client.34145 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9013bcef-adce-49f2-952c-f3f208bf4e96"}]: dispatch 2024-04-27T01:06:52.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:52 smithi181 ceph-mon[30237]: from='client.34145 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9013bcef-adce-49f2-952c-f3f208bf4e96"}]': finished 2024-04-27T01:06:52.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:52 smithi181 ceph-mon[30237]: osdmap e10: 2 total, 1 up, 2 in 2024-04-27T01:06:52.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:52 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:06:52.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4286076989' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:06:54.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:54 smithi162 ceph-mon[26714]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:54.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:54 smithi162 ceph-mon[37680]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:54.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:54 smithi181 ceph-mon[30237]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:56.332 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:56 smithi162 ceph-mon[37680]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:56.332 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:56 smithi162 ceph-mon[26714]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:56.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:56 smithi181 ceph-mon[30237]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:58.309 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:58 smithi162 ceph-mon[26714]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:58.310 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:58 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T01:06:58.310 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:58 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:58.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:58 smithi162 ceph-mon[37680]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:58.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:58 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T01:06:58.310 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:58 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:58.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:58 smithi181 ceph-mon[30237]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:06:58.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:58 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-27T01:06:58.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:58 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:06:59.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:06:59 smithi162 ceph-mon[26714]: Deploying daemon osd.1 on smithi162 2024-04-27T01:06:59.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:06:59 smithi162 ceph-mon[37680]: Deploying daemon osd.1 on smithi162 2024-04-27T01:06:59.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:06:59 smithi181 ceph-mon[30237]: Deploying daemon osd.1 on smithi162 2024-04-27T01:07:00.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:00 smithi162 ceph-mon[26714]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:00.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:00 smithi162 ceph-mon[37680]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:00.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:00 smithi181 ceph-mon[30237]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:02.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:02 smithi162 ceph-mon[26714]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:02.244 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:02 smithi162 ceph-mon[37680]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:02.315 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:02 smithi181 ceph-mon[30237]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:04.036 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:03 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:04.036 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:03 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:04.036 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:03 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:04.036 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:03 smithi162 ceph-mon[26714]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:04.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:03 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:04.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:03 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:04.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:03 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:04.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:03 smithi162 ceph-mon[37680]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:04.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:03 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:04.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:03 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:04.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:03 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:04.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:03 smithi181 ceph-mon[30237]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:06.278 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[37680]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:06.278 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:06.278 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:06.278 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:06.278 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:06.278 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:06.278 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[37680]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T01:07:06.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[26714]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:06.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:06.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:06.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:06.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:06.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:06.279 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:06 smithi162 ceph-mon[26714]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T01:07:06.357 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 1 on host 'smithi162' 2024-04-27T01:07:06.357 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:06.364+0000 7f116e7fc700 1 -- 172.21.15.162:0/1220260657 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 4089390614) 0x7f115c000bc0 con 0x7f116406ef10 2024-04-27T01:07:06.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:06.366+0000 7f117f326700 1 -- 172.21.15.162:0/1220260657 >> v1:172.21.15.162:6800/2166195007 conn(0x7f116406ef10 legacy=0x7f11640713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:06.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:06.366+0000 7f117f326700 1 -- 172.21.15.162:0/1220260657 >> v1:172.21.15.162:6790/0 conn(0x7f117810a1d0 legacy=0x7f11781a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:06.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:06.366+0000 7f117f326700 1 -- 172.21.15.162:0/1220260657 shutdown_connections 2024-04-27T01:07:06.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:06.366+0000 7f117f326700 1 -- 172.21.15.162:0/1220260657 >> 172.21.15.162:0/1220260657 conn(0x7f11780fc550 msgr2=0x7f11780fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:07:06.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:06.366+0000 7f117f326700 1 -- 172.21.15.162:0/1220260657 shutdown_connections 2024-04-27T01:07:06.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:06.366+0000 7f117f326700 1 -- 172.21.15.162:0/1220260657 wait complete. 2024-04-27T01:07:06.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:06 smithi181 ceph-mon[30237]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:06.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:06 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:06.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:06 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:06.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:06 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:06.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:06 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:06.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:06 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:06.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:06 smithi181 ceph-mon[30237]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-27T01:07:07.007 DEBUG:teuthology.orchestra.run.smithi162:osd.1> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.1.service 2024-04-27T01:07:07.010 INFO:tasks.cephadm:Deploying osd.2 on smithi162 with /dev/vg_nvme/lv_2... 2024-04-27T01:07:07.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-27T01:07:07.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:07.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:07.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:07.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[37680]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T01:07:07.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[37680]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T01:07:07.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[37680]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:07.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:07.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[37680]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:07.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:07.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:07.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:07.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[26714]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T01:07:07.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[26714]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T01:07:07.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[26714]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:07.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:07.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:07 smithi162 ceph-mon[26714]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:07.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:07 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:07.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:07 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:07.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:07 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:07.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:07 smithi181 ceph-mon[30237]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-27T01:07:07.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:07 smithi181 ceph-mon[30237]: osdmap e11: 2 total, 1 up, 2 in 2024-04-27T01:07:07.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:07 smithi181 ceph-mon[30237]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:07.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:07 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:07.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:07 smithi181 ceph-mon[30237]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:08.091 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1[47531]: 2024-04-27T01:07:08.009+0000 7fb9d4cbe700 -1 osd.1 0 waiting for initial osdmap 2024-04-27T01:07:08.092 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1[47531]: 2024-04-27T01:07:08.012+0000 7fb9ccaa9700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:07:08.606 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:07:08.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-mon[26714]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:07:08.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-mon[26714]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T01:07:08.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:08.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:08.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-mon[26714]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' 2024-04-27T01:07:08.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-mon[37680]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:07:08.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-mon[37680]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T01:07:08.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:08.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:08.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:08 smithi162 ceph-mon[37680]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' 2024-04-27T01:07:09.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:08 smithi181 ceph-mon[30237]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:07:09.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:08 smithi181 ceph-mon[30237]: osdmap e12: 2 total, 1 up, 2 in 2024-04-27T01:07:09.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:08 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:09.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:08 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:09.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:08 smithi181 ceph-mon[30237]: from='osd.1 v1:172.21.15.162:6805/680367434' entity='osd.1' 2024-04-27T01:07:09.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[26714]: purged_snaps scrub starts 2024-04-27T01:07:09.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[26714]: purged_snaps scrub ok 2024-04-27T01:07:09.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:09.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[26714]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:09.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[26714]: osd.1 v1:172.21.15.162:6805/680367434 boot 2024-04-27T01:07:09.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[26714]: osdmap e13: 2 total, 2 up, 2 in 2024-04-27T01:07:09.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:09.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[37680]: purged_snaps scrub starts 2024-04-27T01:07:09.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[37680]: purged_snaps scrub ok 2024-04-27T01:07:09.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:09.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[37680]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:09.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[37680]: osd.1 v1:172.21.15.162:6805/680367434 boot 2024-04-27T01:07:09.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[37680]: osdmap e13: 2 total, 2 up, 2 in 2024-04-27T01:07:09.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:09 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:10.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:09 smithi181 ceph-mon[30237]: purged_snaps scrub starts 2024-04-27T01:07:10.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:09 smithi181 ceph-mon[30237]: purged_snaps scrub ok 2024-04-27T01:07:10.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:09 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:10.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:09 smithi181 ceph-mon[30237]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-27T01:07:10.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:09 smithi181 ceph-mon[30237]: osd.1 v1:172.21.15.162:6805/680367434 boot 2024-04-27T01:07:10.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:09 smithi181 ceph-mon[30237]: osdmap e13: 2 total, 2 up, 2 in 2024-04-27T01:07:10.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:09 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:07:11.110 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-27T01:07:11.110 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-27T01:07:11.110 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-04-27T01:07:11.110 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-04-27T01:07:11.110 INFO:teuthology.orchestra.run.smithi162.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125396 s, 836 MB/s 2024-04-27T01:07:11.111 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-04-27T01:07:11.424 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: Detected new or changed devices on smithi162 2024-04-27T01:07:11.424 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.424 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.424 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi162 to 8032M 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[26714]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: Detected new or changed devices on smithi162 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:11.425 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi162 to 8032M 2024-04-27T01:07:11.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:11.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:11.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T01:07:11.426 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:11 smithi162 ceph-mon[37680]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:11.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: Detected new or changed devices on smithi162 2024-04-27T01:07:11.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:11.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:11.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi162 to 8032M 2024-04-27T01:07:11.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:11.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:11.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:11.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: osdmap e14: 2 total, 2 up, 2 in 2024-04-27T01:07:11.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:11 smithi181 ceph-mon[30237]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:11.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-04-27T01:07:12.024 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:07:13.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.548+0000 7f9ffee33700 1 -- 172.21.15.162:0/60345396 >> v1:172.21.15.162:6789/0 conn(0x7f9ff8102b70 legacy=0x7f9ff8104f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:13.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.548+0000 7f9ffee33700 1 -- 172.21.15.162:0/60345396 shutdown_connections 2024-04-27T01:07:13.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.548+0000 7f9ffee33700 1 -- 172.21.15.162:0/60345396 >> 172.21.15.162:0/60345396 conn(0x7f9ff80fc500 msgr2=0x7f9ff80fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:07:13.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.548+0000 7f9ffee33700 1 -- 172.21.15.162:0/60345396 shutdown_connections 2024-04-27T01:07:13.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.548+0000 7f9ffee33700 1 -- 172.21.15.162:0/60345396 wait complete. 2024-04-27T01:07:13.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9ffee33700 1 Processor -- start 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9ffee33700 1 -- start start 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9ffee33700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9ff81a46a0 con 0x7f9ff8106680 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9ffee33700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9ff81a58e0 con 0x7f9ff8102b70 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9ffee33700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f9ff81a6b20 con 0x7f9ff810a190 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9ffd3d0700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f9ff810a190 0x7f9ff81a2da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:49820/0 (socket says 172.21.15.162:49820) 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9ffd3d0700 1 -- 172.21.15.162:0/3360162698 learned_addr learned my addr 172.21.15.162:0/3360162698 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9feffff700 1 --1- 172.21.15.162:0/3360162698 >> v1:172.21.15.162:6789/0 conn(0x7f9ff8106680 0x7f9ff819f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:35362/0 (socket says 172.21.15.162:35362) 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2567855203 0 0) 0x7f9ff81a46a0 con 0x7f9ff8106680 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9fe00035b0 con 0x7f9ff8106680 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2595166198 0 0) 0x7f9ff81a6b20 con 0x7f9ff810a190 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ff81a46a0 con 0x7f9ff810a190 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130429151 0 0) 0x7f9ff81a58e0 con 0x7f9ff8102b70 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ff81a6b20 con 0x7f9ff8102b70 2024-04-27T01:07:13.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3694701243 0 0) 0x7f9fe00035b0 con 0x7f9ff8106680 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ff81a58e0 con 0x7f9ff8106680 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1822913976 0 0) 0x7f9ff81a46a0 con 0x7f9ff810a190 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.549+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9fe00035b0 con 0x7f9ff810a190 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f9fe8001130 con 0x7f9ff8106680 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f9ff00017d0 con 0x7f9ff810a190 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4265178039 0 0) 0x7f9ff81a6b20 con 0x7f9ff8102b70 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ff81a46a0 con 0x7f9ff8102b70 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f9ff40048d0 con 0x7f9ff8102b70 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3702322106 0 0) 0x7f9ff81a58e0 con 0x7f9ff8106680 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 >> v1:172.21.15.162:6790/0 conn(0x7f9ff810a190 legacy=0x7f9ff81a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 >> v1:172.21.15.181:6789/0 conn(0x7f9ff8102b70 legacy=0x7f9ff81023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff81a7d60 con 0x7f9ff8106680 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9ffee33700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff81a5b10 con 0x7f9ff8106680 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f9fe8003b10 con 0x7f9ff8106680 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f9fe8005bf0 con 0x7f9ff8106680 2024-04-27T01:07:13.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.550+0000 7f9ffee33700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ff81a6050 con 0x7f9ff8106680 2024-04-27T01:07:13.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.551+0000 7f9ffee33700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff8110cb0 con 0x7f9ff8106680 2024-04-27T01:07:13.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.551+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 14) ==== 92026+0+0 (unknown 428087007 0 0) 0x7f9fe8003680 con 0x7f9ff8106680 2024-04-27T01:07:13.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.552+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(14..14 src has 1..14) ==== 1833+0+0 (unknown 134707008 0 0) 0x7f9fe8088290 con 0x7f9ff8106680 2024-04-27T01:07:13.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.556+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f9fe8056c00 con 0x7f9ff8106680 2024-04-27T01:07:13.754 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:13.763+0000 7f9ffee33700 1 -- 172.21.15.162:0/3360162698 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f9ff81a63b0 con 0x7f9fe006edb0 2024-04-27T01:07:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:14 smithi162 ceph-mon[26714]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:14 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:07:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:14 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:07:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:14 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:14.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:14 smithi162 ceph-mon[37680]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:14.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:14 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:07:14.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:14 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:07:14.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:14 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:14.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:14 smithi181 ceph-mon[30237]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:14.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:14 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:07:14.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:14 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:07:14.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:14 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:15.309 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:15 smithi162 ceph-mon[26714]: from='client.14274 v1:172.21.15.162:0/3360162698' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:15.309 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:15 smithi162 ceph-mon[37680]: from='client.14274 v1:172.21.15.162:0/3360162698' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:15.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:15 smithi181 ceph-mon[30237]: from='client.14274 v1:172.21.15.162:0/3360162698' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:16.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:16 smithi162 ceph-mon[26714]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:16 smithi162 ceph-mon[37680]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:16.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:16 smithi181 ceph-mon[30237]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:17.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/439531086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2212181-797c-4366-b087-2522d5db9e95"}]: dispatch 2024-04-27T01:07:17.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:17 smithi162 ceph-mon[37680]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2212181-797c-4366-b087-2522d5db9e95"}]: dispatch 2024-04-27T01:07:17.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:17 smithi162 ceph-mon[37680]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2212181-797c-4366-b087-2522d5db9e95"}]': finished 2024-04-27T01:07:17.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:17 smithi162 ceph-mon[37680]: osdmap e15: 3 total, 2 up, 3 in 2024-04-27T01:07:17.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:17 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:17.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/439531086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2212181-797c-4366-b087-2522d5db9e95"}]: dispatch 2024-04-27T01:07:17.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:17 smithi162 ceph-mon[26714]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2212181-797c-4366-b087-2522d5db9e95"}]: dispatch 2024-04-27T01:07:17.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:17 smithi162 ceph-mon[26714]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2212181-797c-4366-b087-2522d5db9e95"}]': finished 2024-04-27T01:07:17.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:17 smithi162 ceph-mon[26714]: osdmap e15: 3 total, 2 up, 3 in 2024-04-27T01:07:17.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:17 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:17.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/439531086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2212181-797c-4366-b087-2522d5db9e95"}]: dispatch 2024-04-27T01:07:17.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:17 smithi181 ceph-mon[30237]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2212181-797c-4366-b087-2522d5db9e95"}]: dispatch 2024-04-27T01:07:17.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:17 smithi181 ceph-mon[30237]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2212181-797c-4366-b087-2522d5db9e95"}]': finished 2024-04-27T01:07:17.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:17 smithi181 ceph-mon[30237]: osdmap e15: 3 total, 2 up, 3 in 2024-04-27T01:07:17.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:17 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:18.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:18 smithi162 ceph-mon[37680]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:18.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3251548083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:07:18.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:18 smithi162 ceph-mon[26714]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:18.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3251548083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:07:18.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:18 smithi181 ceph-mon[30237]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:18.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3251548083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:07:20.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:20 smithi162 ceph-mon[26714]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:20.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:20 smithi162 ceph-mon[37680]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:20.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:20 smithi181 ceph-mon[30237]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:22.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:22 smithi162 ceph-mon[26714]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:22.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:22 smithi162 ceph-mon[37680]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:22.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:22 smithi181 ceph-mon[30237]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:24.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:24 smithi162 ceph-mon[26714]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:24.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T01:07:24.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:24.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:24 smithi162 ceph-mon[26714]: Deploying daemon osd.2 on smithi162 2024-04-27T01:07:24.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:24 smithi162 ceph-mon[37680]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:24.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:24 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T01:07:24.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:24 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:24.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:24 smithi162 ceph-mon[37680]: Deploying daemon osd.2 on smithi162 2024-04-27T01:07:24.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:24 smithi181 ceph-mon[30237]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:24.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:24 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-27T01:07:24.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:24 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:24.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:24 smithi181 ceph-mon[30237]: Deploying daemon osd.2 on smithi162 2024-04-27T01:07:26.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:26 smithi162 ceph-mon[26714]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:26.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:26 smithi162 ceph-mon[37680]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:26.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:26 smithi181 ceph-mon[30237]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:28.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:28 smithi162 ceph-mon[26714]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:28.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:28 smithi162 ceph-mon[37680]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:28.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:28 smithi181 ceph-mon[30237]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:29.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:29 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:29.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:29 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:29.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:29 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:29.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:29 smithi162 ceph-mon[26714]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:29.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:29 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:29.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:29 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:29.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:29 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:29.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:29 smithi162 ceph-mon[37680]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:30.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:29 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:30.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:29 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:30.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:29 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:30.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:29 smithi181 ceph-mon[30237]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:32.258 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:32 smithi162 ceph-mon[37680]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:32.259 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:32 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:32.259 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:32 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:32.259 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:32 smithi162 ceph-mon[37680]: from='osd.2 v1:172.21.15.162:6809/2147722624' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T01:07:32.259 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:32 smithi162 ceph-mon[37680]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T01:07:32.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:32 smithi162 ceph-mon[26714]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:32.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:32 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:32.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:32 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:32.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:32 smithi162 ceph-mon[26714]: from='osd.2 v1:172.21.15.162:6809/2147722624' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T01:07:32.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:32 smithi162 ceph-mon[26714]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T01:07:32.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:32.317+0000 7f9fedffb700 1 -- 172.21.15.162:0/3360162698 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 394335690) 0x7f9ff81a63b0 con 0x7f9fe006edb0 2024-04-27T01:07:32.309 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 2 on host 'smithi162' 2024-04-27T01:07:32.312 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:32.321+0000 7f9ffee33700 1 -- 172.21.15.162:0/3360162698 >> v1:172.21.15.162:6800/2166195007 conn(0x7f9fe006edb0 legacy=0x7f9fe0071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:32.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:32.321+0000 7f9ffee33700 1 -- 172.21.15.162:0/3360162698 >> v1:172.21.15.162:6789/0 conn(0x7f9ff8106680 legacy=0x7f9ff819f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:32.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:32.321+0000 7f9ffee33700 1 -- 172.21.15.162:0/3360162698 shutdown_connections 2024-04-27T01:07:32.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:32.321+0000 7f9ffee33700 1 -- 172.21.15.162:0/3360162698 >> 172.21.15.162:0/3360162698 conn(0x7f9ff80fc500 msgr2=0x7f9ff80fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:07:32.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:32.321+0000 7f9ffee33700 1 -- 172.21.15.162:0/3360162698 shutdown_connections 2024-04-27T01:07:32.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:32.321+0000 7f9ffee33700 1 -- 172.21.15.162:0/3360162698 wait complete. 2024-04-27T01:07:32.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:32 smithi181 ceph-mon[30237]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:32.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:32 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:32.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:32 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:32.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:32 smithi181 ceph-mon[30237]: from='osd.2 v1:172.21.15.162:6809/2147722624' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T01:07:32.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:32 smithi181 ceph-mon[30237]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-27T01:07:32.923 DEBUG:teuthology.orchestra.run.smithi162:osd.2> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.2.service 2024-04-27T01:07:32.925 INFO:tasks.cephadm:Deploying osd.3 on smithi162 with /dev/vg_nvme/lv_1... 2024-04-27T01:07:32.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-27T01:07:33.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:33.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:33.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[26714]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T01:07:33.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[26714]: from='osd.2 v1:172.21.15.162:6809/2147722624' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:33.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[26714]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T01:07:33.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:33.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[26714]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:33.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[26714]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:33.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:33.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:33.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[37680]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T01:07:33.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[37680]: from='osd.2 v1:172.21.15.162:6809/2147722624' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:33.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[37680]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T01:07:33.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:33.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[37680]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:33.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:33 smithi162 ceph-mon[37680]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:33.743 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:07:33.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:33 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:33.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:33 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:33.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:33 smithi181 ceph-mon[30237]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-27T01:07:33.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:33 smithi181 ceph-mon[30237]: from='osd.2 v1:172.21.15.162:6809/2147722624' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:33.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:33 smithi181 ceph-mon[30237]: osdmap e16: 3 total, 2 up, 3 in 2024-04-27T01:07:33.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:33 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:33.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:33 smithi181 ceph-mon[30237]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:33.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:33 smithi181 ceph-mon[30237]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:34.091 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2[53457]: 2024-04-27T01:07:34.062+0000 7f687847b700 -1 osd.2 0 waiting for initial osdmap 2024-04-27T01:07:34.091 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2[53457]: 2024-04-27T01:07:34.065+0000 7f686fa65700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:07:34.691 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-mon[26714]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:07:34.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-mon[26714]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T01:07:34.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:34.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:34.692 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-mon[26714]: from='osd.2 ' entity='osd.2' 2024-04-27T01:07:34.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-mon[37680]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:07:34.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-mon[37680]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T01:07:34.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:34.692 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:34.693 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:34 smithi162 ceph-mon[37680]: from='osd.2 ' entity='osd.2' 2024-04-27T01:07:35.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:34 smithi181 ceph-mon[30237]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:07:35.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:34 smithi181 ceph-mon[30237]: osdmap e17: 3 total, 2 up, 3 in 2024-04-27T01:07:35.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:34 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:35.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:34 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:35.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:34 smithi181 ceph-mon[30237]: from='osd.2 ' entity='osd.2' 2024-04-27T01:07:35.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[37680]: purged_snaps scrub starts 2024-04-27T01:07:35.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[37680]: purged_snaps scrub ok 2024-04-27T01:07:35.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:35.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[37680]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:35.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[37680]: osd.2 v1:172.21.15.162:6809/2147722624 boot 2024-04-27T01:07:35.623 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[37680]: osdmap e18: 3 total, 3 up, 3 in 2024-04-27T01:07:35.624 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:35.624 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[26714]: purged_snaps scrub starts 2024-04-27T01:07:35.624 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[26714]: purged_snaps scrub ok 2024-04-27T01:07:35.624 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:35.624 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[26714]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:35.624 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[26714]: osd.2 v1:172.21.15.162:6809/2147722624 boot 2024-04-27T01:07:35.624 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[26714]: osdmap e18: 3 total, 3 up, 3 in 2024-04-27T01:07:35.624 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:35 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:36.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:35 smithi181 ceph-mon[30237]: purged_snaps scrub starts 2024-04-27T01:07:36.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:35 smithi181 ceph-mon[30237]: purged_snaps scrub ok 2024-04-27T01:07:36.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:35 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:36.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:35 smithi181 ceph-mon[30237]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-27T01:07:36.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:35 smithi181 ceph-mon[30237]: osd.2 v1:172.21.15.162:6809/2147722624 boot 2024-04-27T01:07:36.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:35 smithi181 ceph-mon[30237]: osdmap e18: 3 total, 3 up, 3 in 2024-04-27T01:07:36.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:35 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:07:37.019 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-27T01:07:37.020 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-27T01:07:37.020 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-04-27T01:07:37.020 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-04-27T01:07:37.020 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134562 s, 779 MB/s 2024-04-27T01:07:37.020 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-04-27T01:07:37.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[26714]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T01:07:37.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:37.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:37.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:37.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:37.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:37.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:37.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:37.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:37.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:37 smithi162 ceph-mon[37680]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T01:07:37.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-04-27T01:07:37.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:37.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:37.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:07:37.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:37.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:37.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:37.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:37.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:07:37.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:37.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:37 smithi181 ceph-mon[30237]: osdmap e19: 3 total, 3 up, 3 in 2024-04-27T01:07:37.918 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:07:38.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[37680]: Detected new or changed devices on smithi162 2024-04-27T01:07:38.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi162 to 5354M 2024-04-27T01:07:38.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[37680]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:38.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:07:38.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-27T01:07:38.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[37680]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T01:07:38.301 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:07:38.301 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[26714]: Detected new or changed devices on smithi162 2024-04-27T01:07:38.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi162 to 5354M 2024-04-27T01:07:38.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[26714]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:38.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:07:38.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-27T01:07:38.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[26714]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T01:07:38.302 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:38 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:07:38.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:38 smithi181 ceph-mon[30237]: Detected new or changed devices on smithi162 2024-04-27T01:07:38.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:38 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi162 to 5354M 2024-04-27T01:07:38.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:38 smithi181 ceph-mon[30237]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:38.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:38 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:07:38.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:38 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-27T01:07:38.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:38 smithi181 ceph-mon[30237]: osdmap e20: 3 total, 3 up, 3 in 2024-04-27T01:07:38.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:38 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:07:39.183 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:07:38 smithi162 sudo[57325]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:07:39.183 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:07:38 smithi162 sudo[57325]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:39.183 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:07:38 smithi162 sudo[57325]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:39.183 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:07:38 smithi162 sudo[57325]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:39.435 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57333]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:07:39.435 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57333]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:39.435 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57333]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:39.435 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57333]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.465+0000 7f09405be700 1 -- 172.21.15.162:0/1561699666 >> v1:172.21.15.162:6789/0 conn(0x7f0938109040 legacy=0x7f093810b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.465+0000 7f09405be700 1 -- 172.21.15.162:0/1561699666 shutdown_connections 2024-04-27T01:07:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.465+0000 7f09405be700 1 -- 172.21.15.162:0/1561699666 >> 172.21.15.162:0/1561699666 conn(0x7f09380fc500 msgr2=0x7f09380fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:07:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.465+0000 7f09405be700 1 -- 172.21.15.162:0/1561699666 shutdown_connections 2024-04-27T01:07:39.457 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.465+0000 7f09405be700 1 -- 172.21.15.162:0/1561699666 wait complete. 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f09405be700 1 Processor -- start 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f09405be700 1 -- start start 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f09405be700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f093819a6e0 con 0x7f0938109040 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f09405be700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09381a5be0 con 0x7f0938100f10 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f09405be700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f09381a6dc0 con 0x7f09381053b0 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f093db59700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f09381053b0 0x7f093819fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:42816/0 (socket says 172.21.15.162:42816) 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f093db59700 1 -- 172.21.15.162:0/1645345816 learned_addr learned my addr 172.21.15.162:0/1645345816 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684406287 0 0) 0x7f09381a6dc0 con 0x7f09381053b0 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09240035b0 con 0x7f09381053b0 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130938493 0 0) 0x7f09381a5be0 con 0x7f0938100f10 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09381a6dc0 con 0x7f0938100f10 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3707781199 0 0) 0x7f093819a6e0 con 0x7f0938109040 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.466+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09381a5be0 con 0x7f0938109040 2024-04-27T01:07:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1788542396 0 0) 0x7f09240035b0 con 0x7f09381053b0 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f093819a6e0 con 0x7f09381053b0 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f0928001910 con 0x7f09381053b0 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2734048139 0 0) 0x7f09381a6dc0 con 0x7f0938100f10 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09240035b0 con 0x7f0938100f10 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f09340017b0 con 0x7f0938100f10 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 734367226 0 0) 0x7f09381a5be0 con 0x7f0938109040 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f09381a6dc0 con 0x7f0938109040 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f0930001c00 con 0x7f0938109040 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 959847377 0 0) 0x7f093819a6e0 con 0x7f09381053b0 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 >> v1:172.21.15.181:6789/0 conn(0x7f0938100f10 legacy=0x7f0938199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 >> v1:172.21.15.162:6789/0 conn(0x7f0938109040 legacy=0x7f09381a34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09381a7fa0 con 0x7f09381053b0 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f09405be700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f09381a4c30 con 0x7f09381053b0 2024-04-27T01:07:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f09405be700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f09381a5170 con 0x7f09381053b0 2024-04-27T01:07:39.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f0928003b30 con 0x7f09381053b0 2024-04-27T01:07:39.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.467+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f0928005d30 con 0x7f09381053b0 2024-04-27T01:07:39.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.468+0000 7f09405be700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f091c005120 con 0x7f09381053b0 2024-04-27T01:07:39.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.469+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 14) ==== 92026+0+0 (unknown 428087007 0 0) 0x7f09280036a0 con 0x7f09381053b0 2024-04-27T01:07:39.461 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.469+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 2511+0+0 (unknown 1863504429 0 0) 0x7f0928088470 con 0x7f09381053b0 2024-04-27T01:07:39.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.474+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f0928056b00 con 0x7f09381053b0 2024-04-27T01:07:39.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.652+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f09280520c0 con 0x7f09381053b0 2024-04-27T01:07:39.682 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:39.691+0000 7f09405be700 1 -- 172.21.15.162:0/1645345816 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f091c000b90 con 0x7f092406eed0 2024-04-27T01:07:39.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:39 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T01:07:39.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:39 smithi162 ceph-mon[26714]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T01:07:39.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:39 smithi162 ceph-mon[26714]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:39.764 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:39 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:39.764 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57358]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:07:39.764 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57358]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:39.764 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57358]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:39.764 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57358]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:39.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:39 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T01:07:39.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:39 smithi162 ceph-mon[37680]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T01:07:39.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:39 smithi162 ceph-mon[37680]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:39.765 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:39 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:40.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:39 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-27T01:07:40.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:39 smithi181 ceph-mon[30237]: osdmap e21: 3 total, 3 up, 3 in 2024-04-27T01:07:40.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:39 smithi181 ceph-mon[30237]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:40.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:39 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:40.017 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57428]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:07:40.017 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57428]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:40.018 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57428]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:40.018 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:07:39 smithi162 sudo[57428]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:40.292 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57431]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:07:40.292 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57431]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:40.292 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57431]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:40.292 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57431]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:40.544 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57488]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:07:40.544 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57488]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:40.544 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57488]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:40.544 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57488]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:40.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57495]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:07:40.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57495]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:40.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57495]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:40.803 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57495]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:41.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[37680]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-27T01:07:41.061 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[37680]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[37680]: from='client.34199 v1:172.21.15.162:0/1645345816' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57511]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57511]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57511]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:41.062 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 sudo[57511]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:41.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[26714]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-27T01:07:41.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[26714]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T01:07:41.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[26714]: from='client.34199 v1:172.21.15.162:0/1645345816' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:41.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:07:41.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:07:41.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:41.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:07:41.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:07:41.063 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:07:41.063 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:40 smithi181 ceph-mon[30237]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:40 smithi181 ceph-mon[30237]: osdmap e22: 3 total, 3 up, 3 in 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:40 smithi181 ceph-mon[30237]: from='client.34199 v1:172.21.15.162:0/1645345816' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 sudo[32809]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 sudo[32809]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 sudo[32809]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:41.064 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 sudo[32809]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:41.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 sudo[32812]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:07:41.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 sudo[32812]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:41.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 sudo[32812]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:41.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 sudo[32812]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:41.662 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 sudo[57544]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-27T01:07:41.662 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 sudo[57544]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:41.662 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 sudo[57544]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:41.662 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 sudo[57544]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:07:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 sudo[57552]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 sudo[57552]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 sudo[57552]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 sudo[57552]: pam_unix(sudo:session): session closed for user root 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:07:42.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:42.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:07:42.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:07:42.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:41 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:07:42.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:07:42.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:41 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:07:43.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[26714]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:07:43.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[26714]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:07:43.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3856029755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21e98e27-d849-4b8c-9db9-98e738395fd5"}]: dispatch 2024-04-27T01:07:43.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3856029755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21e98e27-d849-4b8c-9db9-98e738395fd5"}]': finished 2024-04-27T01:07:43.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[26714]: osdmap e23: 4 total, 3 up, 4 in 2024-04-27T01:07:43.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:07:43.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[37680]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:07:43.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[37680]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:07:43.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3856029755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21e98e27-d849-4b8c-9db9-98e738395fd5"}]: dispatch 2024-04-27T01:07:43.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3856029755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21e98e27-d849-4b8c-9db9-98e738395fd5"}]': finished 2024-04-27T01:07:43.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[37680]: osdmap e23: 4 total, 3 up, 4 in 2024-04-27T01:07:43.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:42 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:07:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:42 smithi181 ceph-mon[30237]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-27T01:07:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:42 smithi181 ceph-mon[30237]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-27T01:07:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3856029755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21e98e27-d849-4b8c-9db9-98e738395fd5"}]: dispatch 2024-04-27T01:07:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3856029755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21e98e27-d849-4b8c-9db9-98e738395fd5"}]': finished 2024-04-27T01:07:43.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:42 smithi181 ceph-mon[30237]: osdmap e23: 4 total, 3 up, 4 in 2024-04-27T01:07:43.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:42 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:07:44.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:43 smithi181 ceph-mon[30237]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:44.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2075738778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:07:44.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:43 smithi162 ceph-mon[26714]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:44.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2075738778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:07:44.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:43 smithi162 ceph-mon[37680]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:44.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2075738778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:07:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:46 smithi162 ceph-mon[26714]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 147 KiB/s wr, 10 op/s 2024-04-27T01:07:46.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:46 smithi162 ceph-mon[37680]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 147 KiB/s wr, 10 op/s 2024-04-27T01:07:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:46 smithi181 ceph-mon[30237]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 147 KiB/s wr, 10 op/s 2024-04-27T01:07:48.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:48 smithi162 ceph-mon[37680]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 107 KiB/s wr, 7 op/s 2024-04-27T01:07:48.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:48 smithi162 ceph-mon[26714]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 107 KiB/s wr, 7 op/s 2024-04-27T01:07:48.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:48 smithi181 ceph-mon[30237]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 107 KiB/s wr, 7 op/s 2024-04-27T01:07:49.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:49 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T01:07:49.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:49 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:49.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:49 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T01:07:49.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:49 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:49.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:49 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-27T01:07:49.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:49 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:07:50.306 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:50 smithi162 ceph-mon[26714]: Deploying daemon osd.3 on smithi162 2024-04-27T01:07:50.307 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:50 smithi162 ceph-mon[26714]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 84 KiB/s wr, 5 op/s 2024-04-27T01:07:50.307 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:50 smithi162 ceph-mon[37680]: Deploying daemon osd.3 on smithi162 2024-04-27T01:07:50.307 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:50 smithi162 ceph-mon[37680]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 84 KiB/s wr, 5 op/s 2024-04-27T01:07:50.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:50 smithi181 ceph-mon[30237]: Deploying daemon osd.3 on smithi162 2024-04-27T01:07:50.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:50 smithi181 ceph-mon[30237]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 84 KiB/s wr, 5 op/s 2024-04-27T01:07:52.246 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:52 smithi162 ceph-mon[26714]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-27T01:07:52.246 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:52 smithi162 ceph-mon[37680]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-27T01:07:52.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:52 smithi181 ceph-mon[30237]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-27T01:07:54.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:54 smithi162 ceph-mon[26714]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-04-27T01:07:54.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:54 smithi162 ceph-mon[37680]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-04-27T01:07:54.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:54 smithi181 ceph-mon[30237]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-04-27T01:07:55.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:55 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:55.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:55 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:55.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:55 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:55.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:55 smithi181 ceph-mon[30237]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-27T01:07:55.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:55 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:55.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:55 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:55.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:55 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:55.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:55 smithi162 ceph-mon[26714]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-27T01:07:55.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:55 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:55.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:55 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:55.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:55 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:07:55.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:55 smithi162 ceph-mon[37680]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-27T01:07:57.782 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 3 on host 'smithi162' 2024-04-27T01:07:57.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:57.790+0000 7f092f7fe700 1 -- 172.21.15.162:0/1645345816 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 1268054910) 0x7f091c000b90 con 0x7f092406eed0 2024-04-27T01:07:57.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:57.792+0000 7f09405be700 1 -- 172.21.15.162:0/1645345816 >> v1:172.21.15.162:6800/2166195007 conn(0x7f092406eed0 legacy=0x7f0924071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:57.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:57.792+0000 7f09405be700 1 -- 172.21.15.162:0/1645345816 >> v1:172.21.15.162:6790/0 conn(0x7f09381053b0 legacy=0x7f093819fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:07:57.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:57.792+0000 7f09405be700 1 -- 172.21.15.162:0/1645345816 shutdown_connections 2024-04-27T01:07:57.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:57.792+0000 7f09405be700 1 -- 172.21.15.162:0/1645345816 >> 172.21.15.162:0/1645345816 conn(0x7f09380fc500 msgr2=0x7f093806a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:07:57.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:57.792+0000 7f09405be700 1 -- 172.21.15.162:0/1645345816 shutdown_connections 2024-04-27T01:07:57.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:07:57.793+0000 7f09405be700 1 -- 172.21.15.162:0/1645345816 wait complete. 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[37680]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[37680]: from='osd.3 v1:172.21.15.162:6813/2875926639' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[37680]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[26714]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[26714]: from='osd.3 v1:172.21.15.162:6813/2875926639' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[26714]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T01:07:58.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:58 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.402 DEBUG:teuthology.orchestra.run.smithi162:osd.3> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.3.service 2024-04-27T01:07:58.404 INFO:tasks.cephadm:Deploying osd.4 on smithi181 with /dev/vg_nvme/lv_4... 2024-04-27T01:07:58.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-27T01:07:58.431 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:58 smithi181 ceph-mon[30237]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:07:58.431 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:58 smithi181 ceph-mon[30237]: from='osd.3 v1:172.21.15.162:6813/2875926639' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T01:07:58.431 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:58 smithi181 ceph-mon[30237]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-27T01:07:58.431 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:58 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.432 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:58 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.432 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:58 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.432 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:58 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:07:58.648 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:07:59.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-mon[26714]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T01:07:59.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-mon[26714]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T01:07:59.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:07:59.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-mon[26714]: from='osd.3 v1:172.21.15.162:6813/2875926639' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:59.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-mon[26714]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-mon[37680]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T01:07:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-mon[37680]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T01:07:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:07:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-mon[37680]: from='osd.3 v1:172.21.15.162:6813/2875926639' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-mon[37680]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:59.346 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:59 smithi181 ceph-mon[30237]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-27T01:07:59.346 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:59 smithi181 ceph-mon[30237]: osdmap e24: 4 total, 3 up, 4 in 2024-04-27T01:07:59.346 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:59 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:07:59.346 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:59 smithi181 ceph-mon[30237]: from='osd.3 v1:172.21.15.162:6813/2875926639' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:59.346 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:07:59 smithi181 ceph-mon[30237]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-04-27T01:07:59.686 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3[59259]: 2024-04-27T01:07:59.510+0000 7f0d1c68f700 -1 osd.3 0 waiting for initial osdmap 2024-04-27T01:07:59.687 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:07:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3[59259]: 2024-04-27T01:07:59.514+0000 7f0d1447a700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:08:00.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[26714]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:08:00.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[26714]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:08:00.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[26714]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[26714]: from='osd.3 ' entity='osd.3' 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[37680]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[37680]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[37680]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[37680]: from='osd.3 ' entity='osd.3' 2024-04-27T01:08:00.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:00 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:00.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:00 smithi181 ceph-mon[30237]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:08:00.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:00 smithi181 ceph-mon[30237]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-04-27T01:08:00.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:00 smithi181 ceph-mon[30237]: osdmap e25: 4 total, 3 up, 4 in 2024-04-27T01:08:00.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:00 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:00.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:00 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:00.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:00 smithi181 ceph-mon[30237]: from='osd.3 ' entity='osd.3' 2024-04-27T01:08:00.394 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:00 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:00.394 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-27T01:08:00.394 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-27T01:08:00.394 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-04-27T01:08:00.395 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-04-27T01:08:00.395 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121971 s, 860 MB/s 2024-04-27T01:08:00.395 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-04-27T01:08:01.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_4 2024-04-27T01:08:01.331 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:01 smithi181 ceph-mon[30237]: purged_snaps scrub starts 2024-04-27T01:08:01.331 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:01 smithi181 ceph-mon[30237]: purged_snaps scrub ok 2024-04-27T01:08:01.331 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:01 smithi181 ceph-mon[30237]: osd.3 v1:172.21.15.162:6813/2875926639 boot 2024-04-27T01:08:01.331 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:01 smithi181 ceph-mon[30237]: osdmap e26: 4 total, 4 up, 4 in 2024-04-27T01:08:01.331 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:01 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:01.337 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:08:01.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:01 smithi162 ceph-mon[37680]: purged_snaps scrub starts 2024-04-27T01:08:01.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:01 smithi162 ceph-mon[37680]: purged_snaps scrub ok 2024-04-27T01:08:01.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:01 smithi162 ceph-mon[37680]: osd.3 v1:172.21.15.162:6813/2875926639 boot 2024-04-27T01:08:01.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:01 smithi162 ceph-mon[37680]: osdmap e26: 4 total, 4 up, 4 in 2024-04-27T01:08:01.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:01 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:01.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:01 smithi162 ceph-mon[26714]: purged_snaps scrub starts 2024-04-27T01:08:01.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:01 smithi162 ceph-mon[26714]: purged_snaps scrub ok 2024-04-27T01:08:01.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:01 smithi162 ceph-mon[26714]: osd.3 v1:172.21.15.162:6813/2875926639 boot 2024-04-27T01:08:01.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:01 smithi162 ceph-mon[26714]: osdmap e26: 4 total, 4 up, 4 in 2024-04-27T01:08:01.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:01 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:08:02.271 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: Detected new or changed devices on smithi162 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:02.272 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.273 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:02.273 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:02.273 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:02.273 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:02 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: Detected new or changed devices on smithi162 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:02.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: Detected new or changed devices on smithi162 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:02 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:02.949 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.956+0000 7f90435ad700 1 -- 172.21.15.181:0/805352776 >> v1:172.21.15.181:6789/0 conn(0x7f903c100320 legacy=0x7f903c100740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:02.949 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.956+0000 7f90435ad700 1 -- 172.21.15.181:0/805352776 shutdown_connections 2024-04-27T01:08:02.949 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.956+0000 7f90435ad700 1 -- 172.21.15.181:0/805352776 >> 172.21.15.181:0/805352776 conn(0x7f903c0fbca0 msgr2=0x7f903c0fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.956+0000 7f90435ad700 1 -- 172.21.15.181:0/805352776 shutdown_connections 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.956+0000 7f90435ad700 1 -- 172.21.15.181:0/805352776 wait complete. 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90435ad700 1 Processor -- start 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90435ad700 1 -- start start 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90435ad700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f903c1a48e0 con 0x7f903c100320 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90435ad700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f903c1a5b20 con 0x7f903c10a190 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90435ad700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f903c1a6d60 con 0x7f903c106530 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f9041b4a700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f903c10a190 0x7f903c1a2fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:60268/0 (socket says 172.21.15.181:60268) 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f9041b4a700 1 -- 172.21.15.181:0/1172970742 learned_addr learned my addr 172.21.15.181:0/1172970742 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3549820393 0 0) 0x7f903c1a5b20 con 0x7f903c10a190 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90280035b0 con 0x7f903c10a190 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3665650737 0 0) 0x7f903c1a6d60 con 0x7f903c106530 2024-04-27T01:08:02.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f903c1a5b20 con 0x7f903c106530 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769153567 0 0) 0x7f903c1a48e0 con 0x7f903c100320 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f903c1a6d60 con 0x7f903c100320 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 842827009 0 0) 0x7f90280035b0 con 0x7f903c10a190 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f903c1a48e0 con 0x7f903c10a190 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f9034001890 con 0x7f903c10a190 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3431029147 0 0) 0x7f903c1a5b20 con 0x7f903c106530 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f90280035b0 con 0x7f903c106530 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.957+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f902c001830 con 0x7f903c106530 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1175445080 0 0) 0x7f903c1a6d60 con 0x7f903c100320 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f903c1a5b20 con 0x7f903c100320 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f90380017b0 con 0x7f903c100320 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 810456191 0 0) 0x7f903c1a48e0 con 0x7f903c10a190 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 >> v1:172.21.15.162:6790/0 conn(0x7f903c106530 legacy=0x7f903c19f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:02.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 >> v1:172.21.15.162:6789/0 conn(0x7f903c100320 legacy=0x7f903c102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:02.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903c1a7fa0 con 0x7f903c10a190 2024-04-27T01:08:02.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90435ad700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f903c1a6f90 con 0x7f903c10a190 2024-04-27T01:08:02.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90435ad700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f903c1a74d0 con 0x7f903c10a190 2024-04-27T01:08:02.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f9034003b30 con 0x7f903c10a190 2024-04-27T01:08:02.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.958+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f9034005c30 con 0x7f903c10a190 2024-04-27T01:08:02.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.959+0000 7f90435ad700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f903c068b50 con 0x7f903c10a190 2024-04-27T01:08:02.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.959+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f903401c3f0 con 0x7f903c10a190 2024-04-27T01:08:02.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.959+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 2803+0+0 (unknown 98268439 0 0) 0x7f9034088700 con 0x7f903c10a190 2024-04-27T01:08:02.955 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:02.964+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f903405f150 con 0x7f903c10a190 2024-04-27T01:08:03.162 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:03.171+0000 7f90435ad700 1 -- 172.21.15.181:0/1172970742 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f903c1a77c0 con 0x7f902806eeb0 2024-04-27T01:08:03.163 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:03 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi162 to 4016M 2024-04-27T01:08:03.163 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:03 smithi181 ceph-mon[30237]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T01:08:03.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:03 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi162 to 4016M 2024-04-27T01:08:03.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:03 smithi162 ceph-mon[26714]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T01:08:03.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:03 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi162 to 4016M 2024-04-27T01:08:03.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:03 smithi162 ceph-mon[37680]: osdmap e27: 4 total, 4 up, 4 in 2024-04-27T01:08:04.384 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:04 smithi181 ceph-mon[30237]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:04.384 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:04 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:08:04.384 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:04 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:08:04.384 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:04 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:04.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:04 smithi162 ceph-mon[37680]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:04 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:08:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:04 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:08:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:04 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:04.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:04 smithi162 ceph-mon[26714]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:04.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:04 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:08:04.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:04 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:08:04.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:04 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:05.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:05 smithi181 ceph-mon[30237]: from='client.24232 v1:172.21.15.181:0/1172970742' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:05.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:05 smithi162 ceph-mon[26714]: from='client.24232 v1:172.21.15.181:0/1172970742' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:05.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:05 smithi162 ceph-mon[37680]: from='client.24232 v1:172.21.15.181:0/1172970742' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:06.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:06 smithi181 ceph-mon[30237]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:06.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/3652972904' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7cfc95-49e8-440a-8e15-55e6900ae122"}]: dispatch 2024-04-27T01:08:06.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:06 smithi181 ceph-mon[30237]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7cfc95-49e8-440a-8e15-55e6900ae122"}]: dispatch 2024-04-27T01:08:06.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:06 smithi181 ceph-mon[30237]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad7cfc95-49e8-440a-8e15-55e6900ae122"}]': finished 2024-04-27T01:08:06.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:06 smithi181 ceph-mon[30237]: osdmap e28: 5 total, 4 up, 5 in 2024-04-27T01:08:06.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:06 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:06.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[37680]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:06.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/3652972904' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7cfc95-49e8-440a-8e15-55e6900ae122"}]: dispatch 2024-04-27T01:08:06.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[37680]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7cfc95-49e8-440a-8e15-55e6900ae122"}]: dispatch 2024-04-27T01:08:06.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[37680]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad7cfc95-49e8-440a-8e15-55e6900ae122"}]': finished 2024-04-27T01:08:06.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[37680]: osdmap e28: 5 total, 4 up, 5 in 2024-04-27T01:08:06.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:06.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[26714]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:06.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/3652972904' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7cfc95-49e8-440a-8e15-55e6900ae122"}]: dispatch 2024-04-27T01:08:06.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[26714]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad7cfc95-49e8-440a-8e15-55e6900ae122"}]: dispatch 2024-04-27T01:08:06.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[26714]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad7cfc95-49e8-440a-8e15-55e6900ae122"}]': finished 2024-04-27T01:08:06.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[26714]: osdmap e28: 5 total, 4 up, 5 in 2024-04-27T01:08:06.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:06 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:07.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/3237762722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:08:07.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:07 smithi181 ceph-mon[30237]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:07.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/3237762722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:08:07.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:07 smithi162 ceph-mon[26714]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:07.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/3237762722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:08:07.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:07 smithi162 ceph-mon[37680]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:10.145 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:10 smithi181 ceph-mon[30237]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:10.293 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:10 smithi162 ceph-mon[26714]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:10.294 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:10 smithi162 ceph-mon[37680]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:12.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:12 smithi181 ceph-mon[30237]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:12.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:12 smithi162 ceph-mon[26714]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:12.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:12 smithi162 ceph-mon[37680]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:13.215 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T01:08:13.215 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:13.215 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:13 smithi181 ceph-mon[30237]: Deploying daemon osd.4 on smithi181 2024-04-27T01:08:13.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T01:08:13.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:13.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:13 smithi162 ceph-mon[37680]: Deploying daemon osd.4 on smithi181 2024-04-27T01:08:13.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-27T01:08:13.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:13.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:13 smithi162 ceph-mon[26714]: Deploying daemon osd.4 on smithi181 2024-04-27T01:08:14.162 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:14 smithi181 ceph-mon[30237]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:14.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:14 smithi162 ceph-mon[37680]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:14.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:14 smithi162 ceph-mon[26714]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:16.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:16 smithi162 ceph-mon[26714]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:16 smithi162 ceph-mon[37680]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:16.438 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:16 smithi181 ceph-mon[30237]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:18.197 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:18 smithi181 ceph-mon[30237]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:18.197 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:18 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:18.197 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:18 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:18.197 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:18 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:18.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:18 smithi162 ceph-mon[37680]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:18.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:18 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:18.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:18 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:18.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:18 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:18.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:18 smithi162 ceph-mon[26714]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:18.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:18 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:18.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:18 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:18.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:18 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:20.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:20 smithi181 ceph-mon[30237]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:20.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:20 smithi162 ceph-mon[26714]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:20.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:20 smithi162 ceph-mon[37680]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:20.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:20.680+0000 7f90327fc700 1 -- 172.21.15.181:0/1172970742 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 37664100) 0x7f903c1a77c0 con 0x7f902806eeb0 2024-04-27T01:08:20.672 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 4 on host 'smithi181' 2024-04-27T01:08:20.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:20.682+0000 7f90435ad700 1 -- 172.21.15.181:0/1172970742 >> v1:172.21.15.162:6800/2166195007 conn(0x7f902806eeb0 legacy=0x7f9028071370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:20.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:20.682+0000 7f90435ad700 1 -- 172.21.15.181:0/1172970742 >> v1:172.21.15.181:6789/0 conn(0x7f903c10a190 legacy=0x7f903c1a2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:20.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:20.682+0000 7f90435ad700 1 -- 172.21.15.181:0/1172970742 shutdown_connections 2024-04-27T01:08:20.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:20.682+0000 7f90435ad700 1 -- 172.21.15.181:0/1172970742 >> 172.21.15.181:0/1172970742 conn(0x7f903c0fbca0 msgr2=0x7f903c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:08:20.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:20.682+0000 7f90435ad700 1 -- 172.21.15.181:0/1172970742 shutdown_connections 2024-04-27T01:08:20.675 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:20.682+0000 7f90435ad700 1 -- 172.21.15.181:0/1172970742 wait complete. 2024-04-27T01:08:21.310 DEBUG:teuthology.orchestra.run.smithi181:osd.4> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.4.service 2024-04-27T01:08:21.312 INFO:tasks.cephadm:Deploying osd.5 on smithi181 with /dev/vg_nvme/lv_3... 2024-04-27T01:08:21.313 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-27T01:08:21.574 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[26714]: from='osd.4 v1:172.21.15.181:6800/1869417285' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:08:21.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[26714]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:08:21.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.575 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[26714]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:21.575 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.575 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.575 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[37680]: from='osd.4 v1:172.21.15.181:6800/1869417285' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:08:21.576 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[37680]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:08:21.576 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.576 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.576 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:21 smithi162 ceph-mon[37680]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:21.673 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.673 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.673 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:21 smithi181 ceph-mon[30237]: from='osd.4 v1:172.21.15.181:6800/1869417285' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:08:21.673 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:21 smithi181 ceph-mon[30237]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-27T01:08:21.673 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.673 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:21.673 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:21 smithi181 ceph-mon[30237]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:22.231 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:08:22.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:22 smithi162 ceph-mon[37680]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T01:08:22.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:22 smithi162 ceph-mon[37680]: from='osd.4 v1:172.21.15.181:6800/1869417285' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:22.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:22 smithi162 ceph-mon[37680]: osdmap e29: 5 total, 4 up, 5 in 2024-04-27T01:08:22.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:22 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:22.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:22 smithi162 ceph-mon[37680]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:22.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:22 smithi162 ceph-mon[26714]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T01:08:22.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:22 smithi162 ceph-mon[26714]: from='osd.4 v1:172.21.15.181:6800/1869417285' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:22.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:22 smithi162 ceph-mon[26714]: osdmap e29: 5 total, 4 up, 5 in 2024-04-27T01:08:22.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:22 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:22.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:22 smithi162 ceph-mon[26714]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:22.596 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:22 smithi181 ceph-mon[30237]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-27T01:08:22.597 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:22 smithi181 ceph-mon[30237]: from='osd.4 v1:172.21.15.181:6800/1869417285' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:22.597 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:22 smithi181 ceph-mon[30237]: osdmap e29: 5 total, 4 up, 5 in 2024-04-27T01:08:22.597 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:22 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:22.597 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:22 smithi181 ceph-mon[30237]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:22.597 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:08:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:08:22.289+0000 7fdab2f2d700 -1 osd.4 0 waiting for initial osdmap 2024-04-27T01:08:22.597 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:08:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:08:22.294+0000 7fdaaed20700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:08:23.453 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:23 smithi181 ceph-mon[30237]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:08:23.453 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:23 smithi181 ceph-mon[30237]: osdmap e30: 5 total, 4 up, 5 in 2024-04-27T01:08:23.453 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:23.453 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:23.453 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:23 smithi181 ceph-mon[30237]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:23.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:23 smithi162 ceph-mon[37680]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:08:23.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:23 smithi162 ceph-mon[37680]: osdmap e30: 5 total, 4 up, 5 in 2024-04-27T01:08:23.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:23.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:23.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:23 smithi162 ceph-mon[37680]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:23.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:23 smithi162 ceph-mon[26714]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:08:23.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:23 smithi162 ceph-mon[26714]: osdmap e30: 5 total, 4 up, 5 in 2024-04-27T01:08:23.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:23.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:23.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:23 smithi162 ceph-mon[26714]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-27T01:08:24.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[37680]: purged_snaps scrub starts 2024-04-27T01:08:24.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[37680]: purged_snaps scrub ok 2024-04-27T01:08:24.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[37680]: osd.4 v1:172.21.15.181:6800/1869417285 boot 2024-04-27T01:08:24.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[37680]: osdmap e31: 5 total, 5 up, 5 in 2024-04-27T01:08:24.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:24.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[37680]: osdmap e32: 5 total, 5 up, 5 in 2024-04-27T01:08:24.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[26714]: purged_snaps scrub starts 2024-04-27T01:08:24.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[26714]: purged_snaps scrub ok 2024-04-27T01:08:24.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[26714]: osd.4 v1:172.21.15.181:6800/1869417285 boot 2024-04-27T01:08:24.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[26714]: osdmap e31: 5 total, 5 up, 5 in 2024-04-27T01:08:24.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:24.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:24 smithi162 ceph-mon[26714]: osdmap e32: 5 total, 5 up, 5 in 2024-04-27T01:08:24.727 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:24 smithi181 ceph-mon[30237]: purged_snaps scrub starts 2024-04-27T01:08:24.727 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:24 smithi181 ceph-mon[30237]: purged_snaps scrub ok 2024-04-27T01:08:24.727 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:24 smithi181 ceph-mon[30237]: osd.4 v1:172.21.15.181:6800/1869417285 boot 2024-04-27T01:08:24.727 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:24 smithi181 ceph-mon[30237]: osdmap e31: 5 total, 5 up, 5 in 2024-04-27T01:08:24.727 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:24 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:08:24.727 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:24 smithi181 ceph-mon[30237]: osdmap e32: 5 total, 5 up, 5 in 2024-04-27T01:08:25.517 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-27T01:08:25.518 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-27T01:08:25.518 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-04-27T01:08:25.518 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-04-27T01:08:25.518 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123226 s, 851 MB/s 2024-04-27T01:08:25.518 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-04-27T01:08:25.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: Detected new or changed devices on smithi181 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi181 to 17088M 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:25.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: osdmap e33: 5 total, 5 up, 5 in 2024-04-27T01:08:25.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:25 smithi181 ceph-mon[30237]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: Detected new or changed devices on smithi181 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi181 to 17088M 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:25.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:25.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:25.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: osdmap e33: 5 total, 5 up, 5 in 2024-04-27T01:08:25.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[37680]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:25.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: Detected new or changed devices on smithi181 2024-04-27T01:08:25.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:25.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi181 to 17088M 2024-04-27T01:08:25.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:25.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:25.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:25.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:25.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:25.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:25.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: osdmap e33: 5 total, 5 up, 5 in 2024-04-27T01:08:25.843 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:25 smithi162 ceph-mon[26714]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:26.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_3 2024-04-27T01:08:26.377 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:08:27.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:26 smithi181 ceph-mon[30237]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T01:08:27.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:26 smithi162 ceph-mon[37680]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T01:08:27.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:26 smithi162 ceph-mon[26714]: osdmap e34: 5 total, 5 up, 5 in 2024-04-27T01:08:27.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.970+0000 7f7bde2d2700 1 -- 172.21.15.181:0/4150242739 >> v1:172.21.15.181:6789/0 conn(0x7f7bd8106600 legacy=0x7f7bd81089f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:27.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.970+0000 7f7bde2d2700 1 -- 172.21.15.181:0/4150242739 shutdown_connections 2024-04-27T01:08:27.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.970+0000 7f7bde2d2700 1 -- 172.21.15.181:0/4150242739 >> 172.21.15.181:0/4150242739 conn(0x7f7bd80fc500 msgr2=0x7f7bd80fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:08:27.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.970+0000 7f7bde2d2700 1 -- 172.21.15.181:0/4150242739 shutdown_connections 2024-04-27T01:08:27.962 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.970+0000 7f7bde2d2700 1 -- 172.21.15.181:0/4150242739 wait complete. 2024-04-27T01:08:27.963 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.970+0000 7f7bde2d2700 1 Processor -- start 2024-04-27T01:08:27.963 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.970+0000 7f7bde2d2700 1 -- start start 2024-04-27T01:08:27.965 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bde2d2700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7bd81a4930 con 0x7f7bd8102af0 2024-04-27T01:08:27.965 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bde2d2700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7bd81a5b70 con 0x7f7bd810a1a0 2024-04-27T01:08:27.965 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bde2d2700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7bd81a6db0 con 0x7f7bd8106600 2024-04-27T01:08:27.965 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bdc86f700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f7bd810a1a0 0x7f7bd81a3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:36570/0 (socket says 172.21.15.181:36570) 2024-04-27T01:08:27.965 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bdc86f700 1 -- 172.21.15.181:0/1552364878 learned_addr learned my addr 172.21.15.181:0/1552364878 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:08:27.965 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd7fff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f7bd8102af0 0x7f7bd8102370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.181:59280/0 (socket says 172.21.15.181:59280) 2024-04-27T01:08:27.965 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2008034451 0 0) 0x7f7bd81a4930 con 0x7f7bd8102af0 2024-04-27T01:08:27.965 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bc00035b0 con 0x7f7bd8102af0 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2800754845 0 0) 0x7f7bd81a5b70 con 0x7f7bd810a1a0 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bd81a4930 con 0x7f7bd810a1a0 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1105849147 0 0) 0x7f7bd81a6db0 con 0x7f7bd8106600 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7bd81a5b70 con 0x7f7bd8106600 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2338583709 0 0) 0x7f7bd81a4930 con 0x7f7bd810a1a0 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7bd81a6db0 con 0x7f7bd810a1a0 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7bd0001b60 con 0x7f7bd810a1a0 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.971+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1591669678 0 0) 0x7f7bc00035b0 con 0x7f7bd8102af0 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7bd81a4930 con 0x7f7bd8102af0 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7bc80018b0 con 0x7f7bd8102af0 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1280322921 0 0) 0x7f7bd81a5b70 con 0x7f7bd8106600 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7bc00035b0 con 0x7f7bd8106600 2024-04-27T01:08:27.966 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7bcc001430 con 0x7f7bd8106600 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2757602746 0 0) 0x7f7bd81a6db0 con 0x7f7bd810a1a0 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 >> v1:172.21.15.162:6790/0 conn(0x7f7bd8106600 legacy=0x7f7bd819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 >> v1:172.21.15.162:6789/0 conn(0x7f7bd8102af0 legacy=0x7f7bd8102370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd81a7ff0 con 0x7f7bd810a1a0 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bde2d2700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd81a4b60 con 0x7f7bd810a1a0 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bde2d2700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7bd81a50a0 con 0x7f7bd810a1a0 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f7bd0003eb0 con 0x7f7bd810a1a0 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.972+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7bd0006090 con 0x7f7bd810a1a0 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.973+0000 7f7bde2d2700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bc4005150 con 0x7f7bd810a1a0 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.973+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f7bd0003a20 con 0x7f7bd810a1a0 2024-04-27T01:08:27.967 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.974+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(35..35 src has 1..35) ==== 3205+0+0 (unknown 1762158180 0 0) 0x7f7bd0088b20 con 0x7f7bd810a1a0 2024-04-27T01:08:27.970 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:27.979+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f7bd0056e70 con 0x7f7bd810a1a0 2024-04-27T01:08:28.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:27 smithi181 ceph-mon[30237]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T01:08:28.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:27 smithi181 ceph-mon[30237]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:28.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:27 smithi162 ceph-mon[37680]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T01:08:28.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:27 smithi162 ceph-mon[37680]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:28.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:27 smithi162 ceph-mon[26714]: osdmap e35: 5 total, 5 up, 5 in 2024-04-27T01:08:28.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:27 smithi162 ceph-mon[26714]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:28.178 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:28.186+0000 7f7bde2d2700 1 -- 172.21.15.181:0/1552364878 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f7bc4000bc0 con 0x7f7bc006eeb0 2024-04-27T01:08:29.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:28 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:08:29.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:28 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:08:29.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:28 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:29.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:28 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:08:29.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:28 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:08:29.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:28 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:29.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:28 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:08:29.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:28 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:08:29.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:28 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:30.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:29 smithi181 ceph-mon[30237]: from='client.24259 v1:172.21.15.181:0/1552364878' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:30.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:29 smithi181 ceph-mon[30237]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:30.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:29 smithi162 ceph-mon[37680]: from='client.24259 v1:172.21.15.181:0/1552364878' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:30.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:29 smithi162 ceph-mon[37680]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:30.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:29 smithi162 ceph-mon[26714]: from='client.24259 v1:172.21.15.181:0/1552364878' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:30.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:29 smithi162 ceph-mon[26714]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:31.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/3185811612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46341ac9-3fc7-49c0-bc92-03ba76da4ec7"}]: dispatch 2024-04-27T01:08:31.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:30 smithi181 ceph-mon[30237]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46341ac9-3fc7-49c0-bc92-03ba76da4ec7"}]: dispatch 2024-04-27T01:08:31.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:30 smithi181 ceph-mon[30237]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46341ac9-3fc7-49c0-bc92-03ba76da4ec7"}]': finished 2024-04-27T01:08:31.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:30 smithi181 ceph-mon[30237]: osdmap e36: 6 total, 5 up, 6 in 2024-04-27T01:08:31.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:30 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:31.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/3185811612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46341ac9-3fc7-49c0-bc92-03ba76da4ec7"}]: dispatch 2024-04-27T01:08:31.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:30 smithi162 ceph-mon[37680]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46341ac9-3fc7-49c0-bc92-03ba76da4ec7"}]: dispatch 2024-04-27T01:08:31.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:30 smithi162 ceph-mon[37680]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46341ac9-3fc7-49c0-bc92-03ba76da4ec7"}]': finished 2024-04-27T01:08:31.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:30 smithi162 ceph-mon[37680]: osdmap e36: 6 total, 5 up, 6 in 2024-04-27T01:08:31.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:30 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:31.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/3185811612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46341ac9-3fc7-49c0-bc92-03ba76da4ec7"}]: dispatch 2024-04-27T01:08:31.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:30 smithi162 ceph-mon[26714]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46341ac9-3fc7-49c0-bc92-03ba76da4ec7"}]: dispatch 2024-04-27T01:08:31.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:30 smithi162 ceph-mon[26714]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46341ac9-3fc7-49c0-bc92-03ba76da4ec7"}]': finished 2024-04-27T01:08:31.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:30 smithi162 ceph-mon[26714]: osdmap e36: 6 total, 5 up, 6 in 2024-04-27T01:08:31.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:30 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:32.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:31 smithi181 ceph-mon[30237]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-27T01:08:32.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/3256966720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:08:32.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:31 smithi162 ceph-mon[26714]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-27T01:08:32.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/3256966720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:08:32.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:31 smithi162 ceph-mon[37680]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-27T01:08:32.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/3256966720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:08:34.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:34 smithi181 ceph-mon[30237]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-27T01:08:34.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:34 smithi162 ceph-mon[26714]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-27T01:08:34.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:34 smithi162 ceph-mon[37680]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-27T01:08:36.172 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:36 smithi181 ceph-mon[30237]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-27T01:08:36.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:36 smithi162 ceph-mon[26714]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-27T01:08:36.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:36 smithi162 ceph-mon[37680]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-27T01:08:38.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:38 smithi181 ceph-mon[30237]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T01:08:38.131 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:38 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T01:08:38.132 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:38 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:38.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:38 smithi162 ceph-mon[26714]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T01:08:38.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:38 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T01:08:38.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:38 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:38.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:38 smithi162 ceph-mon[37680]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T01:08:38.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:38 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-27T01:08:38.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:38 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:39.129 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:39 smithi181 ceph-mon[30237]: Deploying daemon osd.5 on smithi181 2024-04-27T01:08:39.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:39 smithi162 ceph-mon[26714]: Deploying daemon osd.5 on smithi181 2024-04-27T01:08:39.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:39 smithi162 ceph-mon[37680]: Deploying daemon osd.5 on smithi181 2024-04-27T01:08:40.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:40 smithi162 ceph-mon[37680]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T01:08:40.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:40 smithi162 ceph-mon[26714]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T01:08:40.396 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:40 smithi181 ceph-mon[30237]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T01:08:42.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:42 smithi181 ceph-mon[30237]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:42.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:42 smithi162 ceph-mon[26714]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:42.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:42 smithi162 ceph-mon[37680]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:43.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:43 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:43.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:43 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:43.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:43 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:43.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:43 smithi162 ceph-mon[37680]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:43.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:43 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:43.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:43 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:43.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:43 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:43.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:43 smithi162 ceph-mon[26714]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:43 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:43 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:43 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:43 smithi181 ceph-mon[30237]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:46.160 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:46 smithi181 ceph-mon[30237]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:46.160 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:46.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:46.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:46.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:46.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:46 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:46.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:46 smithi181 ceph-mon[30237]: from='osd.5 v1:172.21.15.181:6804/3172409639' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:08:46.161 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:46 smithi181 ceph-mon[30237]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:08:46.245 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:46.253+0000 7f7bd57fa700 1 -- 172.21.15.181:0/1552364878 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 1579899856) 0x7f7bc4000bc0 con 0x7f7bc006eeb0 2024-04-27T01:08:46.245 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 5 on host 'smithi181' 2024-04-27T01:08:46.246 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:46.256+0000 7f7bde2d2700 1 -- 172.21.15.181:0/1552364878 >> v1:172.21.15.162:6800/2166195007 conn(0x7f7bc006eeb0 legacy=0x7f7bc0071370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:46.247 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:46.256+0000 7f7bde2d2700 1 -- 172.21.15.181:0/1552364878 >> v1:172.21.15.181:6789/0 conn(0x7f7bd810a1a0 legacy=0x7f7bd81a3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:46.247 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:46.256+0000 7f7bde2d2700 1 -- 172.21.15.181:0/1552364878 shutdown_connections 2024-04-27T01:08:46.247 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:46.256+0000 7f7bde2d2700 1 -- 172.21.15.181:0/1552364878 >> 172.21.15.181:0/1552364878 conn(0x7f7bd80fc500 msgr2=0x7f7bd810c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:08:46.247 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:46.256+0000 7f7bde2d2700 1 -- 172.21.15.181:0/1552364878 shutdown_connections 2024-04-27T01:08:46.247 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:46.256+0000 7f7bde2d2700 1 -- 172.21.15.181:0/1552364878 wait complete. 2024-04-27T01:08:46.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[37680]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:46.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:46.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:46.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:46.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:46.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:46.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[37680]: from='osd.5 v1:172.21.15.181:6804/3172409639' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:08:46.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[37680]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:08:46.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[26714]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:46.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:46.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:46.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:46.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:46.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:46.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[26714]: from='osd.5 v1:172.21.15.181:6804/3172409639' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:08:46.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:46 smithi162 ceph-mon[26714]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-27T01:08:46.906 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.5.service 2024-04-27T01:08:46.908 INFO:tasks.cephadm:Deploying osd.6 on smithi181 with /dev/vg_nvme/lv_2... 2024-04-27T01:08:46.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-27T01:08:47.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:47.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:47.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:47.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-mon[30237]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T01:08:47.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-mon[30237]: from='osd.5 v1:172.21.15.181:6804/3172409639' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:47.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-mon[30237]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T01:08:47.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:47.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-mon[30237]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:47.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-mon[30237]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[26714]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T01:08:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[26714]: from='osd.5 v1:172.21.15.181:6804/3172409639' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[26714]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[26714]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[26714]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[37680]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[37680]: from='osd.5 v1:172.21.15.181:6804/3172409639' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[37680]: osdmap e37: 6 total, 5 up, 6 in 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[37680]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:08:47.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:47 smithi162 ceph-mon[37680]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:48.087 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:08:47.825+0000 7f05d376b700 -1 osd.5 0 waiting for initial osdmap 2024-04-27T01:08:48.087 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:08:47 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:08:47.828+0000 7f05cb556700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:08:48.605 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:08:48.728 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:48 smithi181 ceph-mon[30237]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:08:48.728 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:48 smithi181 ceph-mon[30237]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T01:08:48.728 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:48 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:48.728 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:48 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:48.728 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:48 smithi181 ceph-mon[30237]: from='osd.5 ' entity='osd.5' 2024-04-27T01:08:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:48 smithi162 ceph-mon[26714]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:08:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:48 smithi162 ceph-mon[26714]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T01:08:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:48 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:48 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:48 smithi162 ceph-mon[26714]: from='osd.5 ' entity='osd.5' 2024-04-27T01:08:48.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:48 smithi162 ceph-mon[37680]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:08:48.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:48 smithi162 ceph-mon[37680]: osdmap e38: 6 total, 5 up, 6 in 2024-04-27T01:08:48.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:48 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:48.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:48 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:48.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:48 smithi162 ceph-mon[37680]: from='osd.5 ' entity='osd.5' 2024-04-27T01:08:49.639 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:49 smithi181 ceph-mon[30237]: purged_snaps scrub starts 2024-04-27T01:08:49.640 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:49 smithi181 ceph-mon[30237]: purged_snaps scrub ok 2024-04-27T01:08:49.640 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:49 smithi181 ceph-mon[30237]: osd.5 v1:172.21.15.181:6804/3172409639 boot 2024-04-27T01:08:49.640 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:49 smithi181 ceph-mon[30237]: osdmap e39: 6 total, 6 up, 6 in 2024-04-27T01:08:49.640 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:49 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:49.640 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:49 smithi181 ceph-mon[30237]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T01:08:49.640 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:49 smithi181 ceph-mon[30237]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:49.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[26714]: purged_snaps scrub starts 2024-04-27T01:08:49.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[26714]: purged_snaps scrub ok 2024-04-27T01:08:49.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[26714]: osd.5 v1:172.21.15.181:6804/3172409639 boot 2024-04-27T01:08:49.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[26714]: osdmap e39: 6 total, 6 up, 6 in 2024-04-27T01:08:49.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:49.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[26714]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T01:08:49.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[26714]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:49.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[37680]: purged_snaps scrub starts 2024-04-27T01:08:49.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[37680]: purged_snaps scrub ok 2024-04-27T01:08:49.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[37680]: osd.5 v1:172.21.15.181:6804/3172409639 boot 2024-04-27T01:08:49.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[37680]: osdmap e39: 6 total, 6 up, 6 in 2024-04-27T01:08:49.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:08:49.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[37680]: osdmap e40: 6 total, 6 up, 6 in 2024-04-27T01:08:49.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:49 smithi162 ceph-mon[37680]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-27T01:08:50.785 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-27T01:08:50.785 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-27T01:08:50.785 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-04-27T01:08:50.785 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-04-27T01:08:50.785 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134632 s, 779 MB/s 2024-04-27T01:08:50.785 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-04-27T01:08:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:50 smithi181 ceph-mon[30237]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T01:08:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:50 smithi162 ceph-mon[26714]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T01:08:51.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:50 smithi162 ceph-mon[37680]: osdmap e41: 6 total, 6 up, 6 in 2024-04-27T01:08:51.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_2 2024-04-27T01:08:51.693 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:08:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T01:08:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: Detected new or changed devices on smithi181 2024-04-27T01:08:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi181 to 8544M 2024-04-27T01:08:52.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:52.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:52.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:08:52.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:51 smithi181 ceph-mon[30237]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T01:08:52.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: Detected new or changed devices on smithi181 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi181 to 8544M 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:08:52.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[37680]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: osdmap e42: 6 total, 6 up, 6 in 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: Detected new or changed devices on smithi181 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi181 to 8544M 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:08:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: pgmap v137: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:08:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:51 smithi162 ceph-mon[26714]: osdmap e43: 6 total, 6 up, 6 in 2024-04-27T01:08:52.938 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:52 smithi181 ceph-mon[30237]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T01:08:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:52 smithi162 ceph-mon[37680]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T01:08:53.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:52 smithi162 ceph-mon[26714]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-27T01:08:53.146 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.155+0000 7f607ec9e700 1 -- 172.21.15.181:0/2863569362 >> v1:172.21.15.181:6789/0 conn(0x7f60781064a0 legacy=0x7f60781088f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:53.146 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.155+0000 7f607ec9e700 1 -- 172.21.15.181:0/2863569362 shutdown_connections 2024-04-27T01:08:53.146 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.155+0000 7f607ec9e700 1 -- 172.21.15.181:0/2863569362 >> 172.21.15.181:0/2863569362 conn(0x7f60780fd630 msgr2=0x7f60780ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:08:53.146 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.155+0000 7f607ec9e700 1 -- 172.21.15.181:0/2863569362 shutdown_connections 2024-04-27T01:08:53.146 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.155+0000 7f607ec9e700 1 -- 172.21.15.181:0/2863569362 wait complete. 2024-04-27T01:08:53.148 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f607ec9e700 1 Processor -- start 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f607ec9e700 1 -- start start 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f607ec9e700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f60781981d0 con 0x7f607810a1e0 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f607ec9e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f60781a39c0 con 0x7f60781064a0 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f607ec9e700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f60781a4ba0 con 0x7f6078102070 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f6077fff700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f60781064a0 0x7f607819db60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:37088/0 (socket says 172.21.15.181:37088) 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f6077fff700 1 -- 172.21.15.181:0/3915638426 learned_addr learned my addr 172.21.15.181:0/3915638426 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1827995327 0 0) 0x7f60781a39c0 con 0x7f60781064a0 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60680035b0 con 0x7f60781064a0 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3117850790 0 0) 0x7f60781a4ba0 con 0x7f6078102070 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60781a39c0 con 0x7f6078102070 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 406028458 0 0) 0x7f60781981d0 con 0x7f607810a1e0 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.156+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60781a4ba0 con 0x7f607810a1e0 2024-04-27T01:08:53.149 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3693397276 0 0) 0x7f60680035b0 con 0x7f60781064a0 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60781981d0 con 0x7f60781064a0 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6064001430 con 0x7f60781064a0 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4176058082 0 0) 0x7f60781a39c0 con 0x7f6078102070 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60680035b0 con 0x7f6078102070 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f606c0017e0 con 0x7f6078102070 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2407785444 0 0) 0x7f60781a4ba0 con 0x7f607810a1e0 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60781a39c0 con 0x7f607810a1e0 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6070001af0 con 0x7f607810a1e0 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1001869071 0 0) 0x7f60781981d0 con 0x7f60781064a0 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 >> v1:172.21.15.162:6790/0 conn(0x7f6078102070 legacy=0x7f6078197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 >> v1:172.21.15.162:6789/0 conn(0x7f607810a1e0 legacy=0x7f60781a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60781a5dc0 con 0x7f60781064a0 2024-04-27T01:08:53.150 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f607ec9e700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60781a2a10 con 0x7f60781064a0 2024-04-27T01:08:53.151 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f60640016e0 con 0x7f60781064a0 2024-04-27T01:08:53.151 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6064005190 con 0x7f60781064a0 2024-04-27T01:08:53.151 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.157+0000 7f607ec9e700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f60781a2f50 con 0x7f60781064a0 2024-04-27T01:08:53.151 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.158+0000 7f607ec9e700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6078068b50 con 0x7f60781064a0 2024-04-27T01:08:53.151 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.158+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f6064005410 con 0x7f60781064a0 2024-04-27T01:08:53.151 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.159+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(43..43 src has 1..43) ==== 3497+0+0 (unknown 1300407386 0 0) 0x7f6064088c90 con 0x7f60781064a0 2024-04-27T01:08:53.154 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.163+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f6064056e60 con 0x7f60781064a0 2024-04-27T01:08:53.360 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:08:53.369+0000 7f607ec9e700 1 -- 172.21.15.181:0/3915638426 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f60781a32b0 con 0x7f606806eee0 2024-04-27T01:08:54.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:53 smithi181 ceph-mon[30237]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:08:54.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:53 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:08:54.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:53 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:08:54.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:53 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:53 smithi162 ceph-mon[26714]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:08:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:53 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:08:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:53 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:08:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:53 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:54.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:53 smithi162 ceph-mon[37680]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:08:54.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:53 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:08:54.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:53 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:08:54.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:53 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:08:55.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:54 smithi181 ceph-mon[30237]: from='client.24286 v1:172.21.15.181:0/3915638426' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:55.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:54 smithi162 ceph-mon[26714]: from='client.24286 v1:172.21.15.181:0/3915638426' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:55.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:54 smithi162 ceph-mon[37680]: from='client.24286 v1:172.21.15.181:0/3915638426' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:08:56.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:55 smithi162 ceph-mon[26714]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:08:56.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:55 smithi162 ceph-mon[37680]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:08:56.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:55 smithi181 ceph-mon[30237]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:08:57.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:56 smithi181 ceph-mon[30237]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-27T01:08:57.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:56 smithi181 ceph-mon[30237]: Cluster is now healthy 2024-04-27T01:08:57.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/2073939393' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "832670c0-ca6f-484e-8719-3cf680f03fb8"}]: dispatch 2024-04-27T01:08:57.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:56 smithi181 ceph-mon[30237]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "832670c0-ca6f-484e-8719-3cf680f03fb8"}]: dispatch 2024-04-27T01:08:57.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:56 smithi181 ceph-mon[30237]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "832670c0-ca6f-484e-8719-3cf680f03fb8"}]': finished 2024-04-27T01:08:57.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:56 smithi181 ceph-mon[30237]: osdmap e44: 7 total, 6 up, 7 in 2024-04-27T01:08:57.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:56 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:08:57.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/1184977570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:08:57.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[26714]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-27T01:08:57.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[26714]: Cluster is now healthy 2024-04-27T01:08:57.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/2073939393' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "832670c0-ca6f-484e-8719-3cf680f03fb8"}]: dispatch 2024-04-27T01:08:57.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[26714]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "832670c0-ca6f-484e-8719-3cf680f03fb8"}]: dispatch 2024-04-27T01:08:57.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[26714]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "832670c0-ca6f-484e-8719-3cf680f03fb8"}]': finished 2024-04-27T01:08:57.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[26714]: osdmap e44: 7 total, 6 up, 7 in 2024-04-27T01:08:57.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:08:57.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/1184977570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:08:57.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[37680]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-27T01:08:57.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[37680]: Cluster is now healthy 2024-04-27T01:08:57.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/2073939393' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "832670c0-ca6f-484e-8719-3cf680f03fb8"}]: dispatch 2024-04-27T01:08:57.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[37680]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "832670c0-ca6f-484e-8719-3cf680f03fb8"}]: dispatch 2024-04-27T01:08:57.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[37680]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "832670c0-ca6f-484e-8719-3cf680f03fb8"}]': finished 2024-04-27T01:08:57.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[37680]: osdmap e44: 7 total, 6 up, 7 in 2024-04-27T01:08:57.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:08:57.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/1184977570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:08:58.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:08:57 smithi162 ceph-mon[26714]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-27T01:08:58.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:08:57 smithi162 ceph-mon[37680]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-27T01:08:58.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:08:57 smithi181 ceph-mon[30237]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-27T01:09:00.334 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:00 smithi181 ceph-mon[30237]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T01:09:00.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:00 smithi162 ceph-mon[26714]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T01:09:00.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:00 smithi162 ceph-mon[37680]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T01:09:02.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:02 smithi162 ceph-mon[26714]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-27T01:09:02.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:02 smithi162 ceph-mon[37680]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-27T01:09:02.462 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:02 smithi181 ceph-mon[30237]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-27T01:09:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:03 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T01:09:03.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:03 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:03.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:03 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T01:09:03.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:03 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:03.364 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:03 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-27T01:09:03.365 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:03 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:04.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:04 smithi162 ceph-mon[26714]: Deploying daemon osd.6 on smithi181 2024-04-27T01:09:04.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:04 smithi162 ceph-mon[26714]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T01:09:04.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:04 smithi162 ceph-mon[37680]: Deploying daemon osd.6 on smithi181 2024-04-27T01:09:04.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:04 smithi162 ceph-mon[37680]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T01:09:04.383 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:04 smithi181 ceph-mon[30237]: Deploying daemon osd.6 on smithi181 2024-04-27T01:09:04.384 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:04 smithi181 ceph-mon[30237]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-27T01:09:06.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:06 smithi162 ceph-mon[26714]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:06.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:06 smithi162 ceph-mon[37680]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:06.421 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:06 smithi181 ceph-mon[30237]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:08.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:08 smithi181 ceph-mon[30237]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:08.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:08 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:08.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:08 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:08.333 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:08 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:08.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:08 smithi162 ceph-mon[26714]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:08.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:08 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:08.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:08 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:08.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:08 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:08.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:08 smithi162 ceph-mon[37680]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:08.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:08 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:08.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:08 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:08.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:08 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:10.175 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:10 smithi181 ceph-mon[30237]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:10.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:10 smithi162 ceph-mon[37680]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:10.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:10 smithi162 ceph-mon[26714]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:11.521 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:11.530+0000 7f6075ffb700 1 -- 172.21.15.181:0/3915638426 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 3121965068) 0x7f60781a32b0 con 0x7f606806eee0 2024-04-27T01:09:11.522 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 6 on host 'smithi181' 2024-04-27T01:09:11.523 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:11.532+0000 7f607ec9e700 1 -- 172.21.15.181:0/3915638426 >> v1:172.21.15.162:6800/2166195007 conn(0x7f606806eee0 legacy=0x7f60680713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:11.523 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:11.532+0000 7f607ec9e700 1 -- 172.21.15.181:0/3915638426 >> v1:172.21.15.181:6789/0 conn(0x7f60781064a0 legacy=0x7f607819db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:11.523 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:11.532+0000 7f607ec9e700 1 -- 172.21.15.181:0/3915638426 shutdown_connections 2024-04-27T01:09:11.524 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:11.532+0000 7f607ec9e700 1 -- 172.21.15.181:0/3915638426 >> 172.21.15.181:0/3915638426 conn(0x7f60780fd630 msgr2=0x7f60781088f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:11.524 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:11.532+0000 7f607ec9e700 1 -- 172.21.15.181:0/3915638426 shutdown_connections 2024-04-27T01:09:11.524 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:11.532+0000 7f607ec9e700 1 -- 172.21.15.181:0/3915638426 wait complete. 2024-04-27T01:09:12.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:12.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:12.095 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.096 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:12.096 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: from='osd.6 v1:172.21.15.181:6808/30247032' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:09:12.096 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:09:12.096 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.096 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.096 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:11 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:12.097 DEBUG:teuthology.orchestra.run.smithi181:osd.6> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.6.service 2024-04-27T01:09:12.099 INFO:tasks.cephadm:Deploying osd.7 on smithi181 with /dev/vg_nvme/lv_1... 2024-04-27T01:09:12.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: from='osd.6 v1:172.21.15.181:6808/30247032' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: from='osd.6 v1:172.21.15.181:6808/30247032' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:12.343 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:11 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:13.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:12 smithi181 ceph-mon[30237]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T01:09:13.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:12 smithi181 ceph-mon[30237]: from='osd.6 v1:172.21.15.181:6808/30247032' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:13.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:12 smithi181 ceph-mon[30237]: osdmap e45: 7 total, 6 up, 7 in 2024-04-27T01:09:13.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:12 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:13.105 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:12 smithi181 ceph-mon[30237]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:13.143 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:09:13.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:12 smithi162 ceph-mon[37680]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T01:09:13.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:12 smithi162 ceph-mon[37680]: from='osd.6 v1:172.21.15.181:6808/30247032' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:13.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:12 smithi162 ceph-mon[37680]: osdmap e45: 7 total, 6 up, 7 in 2024-04-27T01:09:13.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:12 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:13.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:12 smithi162 ceph-mon[37680]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:13.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:12 smithi162 ceph-mon[26714]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-27T01:09:13.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:12 smithi162 ceph-mon[26714]: from='osd.6 v1:172.21.15.181:6808/30247032' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:13.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:12 smithi162 ceph-mon[26714]: osdmap e45: 7 total, 6 up, 7 in 2024-04-27T01:09:13.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:12 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:13.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:12 smithi162 ceph-mon[26714]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:13.414 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:09:13.343+0000 7f7f693b7700 -1 osd.6 0 waiting for initial osdmap 2024-04-27T01:09:13.414 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:09:13.349+0000 7f7f609a1700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:09:14.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-mon[30237]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:09:14.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-mon[30237]: osdmap e46: 7 total, 6 up, 7 in 2024-04-27T01:09:14.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:14.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:14.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-mon[30237]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:14.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-mon[30237]: from='osd.6 ' entity='osd.6' 2024-04-27T01:09:14.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-mon[30237]: osd.6 v1:172.21.15.181:6808/30247032 boot 2024-04-27T01:09:14.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-mon[30237]: osdmap e47: 7 total, 7 up, 7 in 2024-04-27T01:09:14.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:13 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:14.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[37680]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:09:14.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[37680]: osdmap e46: 7 total, 6 up, 7 in 2024-04-27T01:09:14.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:14.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:14.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[37680]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:14.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[37680]: from='osd.6 ' entity='osd.6' 2024-04-27T01:09:14.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[37680]: osd.6 v1:172.21.15.181:6808/30247032 boot 2024-04-27T01:09:14.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[37680]: osdmap e47: 7 total, 7 up, 7 in 2024-04-27T01:09:14.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[26714]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:09:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[26714]: osdmap e46: 7 total, 6 up, 7 in 2024-04-27T01:09:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[26714]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-27T01:09:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[26714]: from='osd.6 ' entity='osd.6' 2024-04-27T01:09:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[26714]: osd.6 v1:172.21.15.181:6808/30247032 boot 2024-04-27T01:09:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[26714]: osdmap e47: 7 total, 7 up, 7 in 2024-04-27T01:09:14.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:13 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:09:14.997 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:14 smithi181 ceph-mon[30237]: purged_snaps scrub starts 2024-04-27T01:09:14.997 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:14 smithi181 ceph-mon[30237]: purged_snaps scrub ok 2024-04-27T01:09:14.997 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:14 smithi181 ceph-mon[30237]: osdmap e48: 7 total, 7 up, 7 in 2024-04-27T01:09:15.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:14 smithi162 ceph-mon[37680]: purged_snaps scrub starts 2024-04-27T01:09:15.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:14 smithi162 ceph-mon[37680]: purged_snaps scrub ok 2024-04-27T01:09:15.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:14 smithi162 ceph-mon[37680]: osdmap e48: 7 total, 7 up, 7 in 2024-04-27T01:09:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:14 smithi162 ceph-mon[26714]: purged_snaps scrub starts 2024-04-27T01:09:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:14 smithi162 ceph-mon[26714]: purged_snaps scrub ok 2024-04-27T01:09:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:14 smithi162 ceph-mon[26714]: osdmap e48: 7 total, 7 up, 7 in 2024-04-27T01:09:15.729 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-27T01:09:15.729 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-27T01:09:15.730 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2024-04-27T01:09:15.730 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2024-04-27T01:09:15.730 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124726 s, 841 MB/s 2024-04-27T01:09:15.730 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2024-04-27T01:09:16.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-04-27T01:09:16.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: osdmap e49: 7 total, 7 up, 7 in 2024-04-27T01:09:16.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:16.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:16.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:16.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:16.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:16.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:15 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-04-27T01:09:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: osdmap e49: 7 total, 7 up, 7 in 2024-04-27T01:09:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: osdmap e49: 7 total, 7 up, 7 in 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:16.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:16.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:16.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:15 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:16.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi181:vg_nvme/lv_1 2024-04-27T01:09:16.579 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:09:16.952 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:16 smithi181 ceph-mon[30237]: Detected new or changed devices on smithi181 2024-04-27T01:09:16.952 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:16 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi181 to 5696M 2024-04-27T01:09:17.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:16 smithi162 ceph-mon[37680]: Detected new or changed devices on smithi181 2024-04-27T01:09:17.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:16 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi181 to 5696M 2024-04-27T01:09:17.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:16 smithi162 ceph-mon[26714]: Detected new or changed devices on smithi181 2024-04-27T01:09:17.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:16 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi181 to 5696M 2024-04-27T01:09:18.072 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:17 smithi181 ceph-mon[30237]: osdmap e50: 7 total, 7 up, 7 in 2024-04-27T01:09:18.073 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:17 smithi181 ceph-mon[30237]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-04-27T01:09:18.177 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.185+0000 7f4f33b9a700 1 -- 172.21.15.181:0/2376227292 >> v1:172.21.15.181:6789/0 conn(0x7f4f2c102550 legacy=0x7f4f2c102970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:18.177 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f33b9a700 1 -- 172.21.15.181:0/2376227292 shutdown_connections 2024-04-27T01:09:18.177 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f33b9a700 1 -- 172.21.15.181:0/2376227292 >> 172.21.15.181:0/2376227292 conn(0x7f4f2c0fbd00 msgr2=0x7f4f2c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:18.177 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f33b9a700 1 -- 172.21.15.181:0/2376227292 shutdown_connections 2024-04-27T01:09:18.178 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f33b9a700 1 -- 172.21.15.181:0/2376227292 wait complete. 2024-04-27T01:09:18.178 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f33b9a700 1 Processor -- start 2024-04-27T01:09:18.178 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f33b9a700 1 -- start start 2024-04-27T01:09:18.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f33b9a700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4f2c1973c0 con 0x7f4f2c102550 2024-04-27T01:09:18.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f33b9a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4f2c1a2bb0 con 0x7f4f2c1053f0 2024-04-27T01:09:18.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f33b9a700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4f2c1a3d90 con 0x7f4f2c1090c0 2024-04-27T01:09:18.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f31135700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f4f2c1053f0 0x7f4f2c19cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:38068/0 (socket says 172.21.15.181:38068) 2024-04-27T01:09:18.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.186+0000 7f4f31135700 1 -- 172.21.15.181:0/199435298 learned_addr learned my addr 172.21.15.181:0/199435298 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:09:18.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3007341981 0 0) 0x7f4f2c1a2bb0 con 0x7f4f2c1053f0 2024-04-27T01:09:18.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f180035b0 con 0x7f4f2c1053f0 2024-04-27T01:09:18.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3003220730 0 0) 0x7f4f2c1a3d90 con 0x7f4f2c1090c0 2024-04-27T01:09:18.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f2c1a2bb0 con 0x7f4f2c1090c0 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2112723228 0 0) 0x7f4f2c1973c0 con 0x7f4f2c102550 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f2c1a3d90 con 0x7f4f2c102550 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1979047902 0 0) 0x7f4f180035b0 con 0x7f4f2c1053f0 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f2c1973c0 con 0x7f4f2c1053f0 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f4f280018b0 con 0x7f4f2c1053f0 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 731658994 0 0) 0x7f4f2c1a3d90 con 0x7f4f2c102550 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f180035b0 con 0x7f4f2c102550 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f4f1c001430 con 0x7f4f2c102550 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3662680432 0 0) 0x7f4f2c1a2bb0 con 0x7f4f2c1090c0 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f2c1a3d90 con 0x7f4f2c1090c0 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f4f24001af0 con 0x7f4f2c1090c0 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3480375115 0 0) 0x7f4f2c1973c0 con 0x7f4f2c1053f0 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 >> v1:172.21.15.162:6790/0 conn(0x7f4f2c1090c0 legacy=0x7f4f2c1a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 >> v1:172.21.15.162:6789/0 conn(0x7f4f2c102550 legacy=0x7f4f2c196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:18.181 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2c1a4fb0 con 0x7f4f2c1053f0 2024-04-27T01:09:18.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f33b9a700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4f2c1a3fc0 con 0x7f4f2c1053f0 2024-04-27T01:09:18.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f33b9a700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4f2c1a4500 con 0x7f4f2c1053f0 2024-04-27T01:09:18.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f4f28001ca0 con 0x7f4f2c1053f0 2024-04-27T01:09:18.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.187+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f4f28005c90 con 0x7f4f2c1053f0 2024-04-27T01:09:18.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.188+0000 7f4f33b9a700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f10005150 con 0x7f4f2c1053f0 2024-04-27T01:09:18.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.189+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f4f28003950 con 0x7f4f2c1053f0 2024-04-27T01:09:18.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.189+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(51..51 src has 1..51) ==== 3789+0+0 (unknown 3404217212 0 0) 0x7f4f28088910 con 0x7f4f2c1053f0 2024-04-27T01:09:18.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.194+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f4f28056a20 con 0x7f4f2c1053f0 2024-04-27T01:09:18.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:17 smithi162 ceph-mon[37680]: osdmap e50: 7 total, 7 up, 7 in 2024-04-27T01:09:18.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:17 smithi162 ceph-mon[37680]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-04-27T01:09:18.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:17 smithi162 ceph-mon[26714]: osdmap e50: 7 total, 7 up, 7 in 2024-04-27T01:09:18.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:17 smithi162 ceph-mon[26714]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 349 MiB used, 625 GiB / 626 GiB avail 2024-04-27T01:09:18.389 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:18.398+0000 7f4f33b9a700 1 -- 172.21.15.181:0/199435298 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f4f10000bc0 con 0x7f4f18071060 2024-04-27T01:09:19.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:18 smithi181 ceph-mon[30237]: osdmap e51: 7 total, 7 up, 7 in 2024-04-27T01:09:19.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:18 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:09:19.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:18 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:09:19.234 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:18 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:19.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:18 smithi162 ceph-mon[37680]: osdmap e51: 7 total, 7 up, 7 in 2024-04-27T01:09:19.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:18 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:09:19.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:18 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:09:19.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:18 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:18 smithi162 ceph-mon[26714]: osdmap e51: 7 total, 7 up, 7 in 2024-04-27T01:09:19.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:18 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-27T01:09:19.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:18 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-27T01:09:19.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:18 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:20.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:19 smithi181 ceph-mon[30237]: from='client.24313 v1:172.21.15.181:0/199435298' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:20.135 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:19 smithi181 ceph-mon[30237]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-04-27T01:09:20.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:19 smithi162 ceph-mon[37680]: from='client.24313 v1:172.21.15.181:0/199435298' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:20.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:19 smithi162 ceph-mon[37680]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-04-27T01:09:20.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:19 smithi162 ceph-mon[26714]: from='client.24313 v1:172.21.15.181:0/199435298' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:20.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:19 smithi162 ceph-mon[26714]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-04-27T01:09:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/1548565630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63a81b98-465c-4a37-a666-90133708abc4"}]: dispatch 2024-04-27T01:09:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:20 smithi181 ceph-mon[30237]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63a81b98-465c-4a37-a666-90133708abc4"}]: dispatch 2024-04-27T01:09:21.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:20 smithi181 ceph-mon[30237]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63a81b98-465c-4a37-a666-90133708abc4"}]': finished 2024-04-27T01:09:21.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/1548565630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63a81b98-465c-4a37-a666-90133708abc4"}]: dispatch 2024-04-27T01:09:21.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:20 smithi162 ceph-mon[37680]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63a81b98-465c-4a37-a666-90133708abc4"}]: dispatch 2024-04-27T01:09:21.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:20 smithi162 ceph-mon[37680]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63a81b98-465c-4a37-a666-90133708abc4"}]': finished 2024-04-27T01:09:21.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/1548565630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63a81b98-465c-4a37-a666-90133708abc4"}]: dispatch 2024-04-27T01:09:21.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:20 smithi162 ceph-mon[26714]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63a81b98-465c-4a37-a666-90133708abc4"}]: dispatch 2024-04-27T01:09:21.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:20 smithi162 ceph-mon[26714]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63a81b98-465c-4a37-a666-90133708abc4"}]': finished 2024-04-27T01:09:22.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:21 smithi181 ceph-mon[30237]: osdmap e52: 8 total, 7 up, 8 in 2024-04-27T01:09:22.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:22.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:21 smithi181 ceph-mon[30237]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-27T01:09:22.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/2359538089' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:09:22.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:21 smithi162 ceph-mon[26714]: osdmap e52: 8 total, 7 up, 8 in 2024-04-27T01:09:22.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:22.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:21 smithi162 ceph-mon[26714]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-27T01:09:22.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/2359538089' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:09:22.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:21 smithi162 ceph-mon[37680]: osdmap e52: 8 total, 7 up, 8 in 2024-04-27T01:09:22.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:22.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:21 smithi162 ceph-mon[37680]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-27T01:09:22.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/2359538089' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-27T01:09:24.162 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:24 smithi181 ceph-mon[30237]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-27T01:09:24.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:24 smithi162 ceph-mon[26714]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-27T01:09:24.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:24 smithi162 ceph-mon[37680]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-27T01:09:26.310 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:26 smithi181 ceph-mon[30237]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T01:09:26.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:26 smithi162 ceph-mon[26714]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T01:09:26.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:26 smithi162 ceph-mon[37680]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T01:09:28.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:28 smithi181 ceph-mon[30237]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-27T01:09:28.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:28 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T01:09:28.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:28 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:28.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:28 smithi162 ceph-mon[26714]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-27T01:09:28.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:28 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T01:09:28.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:28 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:28.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:28 smithi162 ceph-mon[37680]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-27T01:09:28.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:28 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-27T01:09:28.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:28 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:29.309 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:29 smithi181 ceph-mon[30237]: Deploying daemon osd.7 on smithi181 2024-04-27T01:09:29.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:29 smithi162 ceph-mon[26714]: Deploying daemon osd.7 on smithi181 2024-04-27T01:09:29.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:29 smithi162 ceph-mon[37680]: Deploying daemon osd.7 on smithi181 2024-04-27T01:09:30.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:30 smithi181 ceph-mon[30237]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:30.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:30 smithi162 ceph-mon[26714]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:30.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:30 smithi162 ceph-mon[37680]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:32.150 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:32 smithi181 ceph-mon[30237]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:32.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:32 smithi162 ceph-mon[37680]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:32.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:32 smithi162 ceph-mon[26714]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:33.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:33.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:33.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:33 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:33.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:33 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:33.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:33 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:33.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:33 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:33.436 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:33 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:33.436 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:33 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:33.436 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:33 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:34.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:34 smithi162 ceph-mon[26714]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:34.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:34 smithi162 ceph-mon[37680]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:34.456 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:34 smithi181 ceph-mon[30237]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:36.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:36 smithi181 ceph-mon[30237]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:36.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:36 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:36.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:36 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:36.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:36 smithi181 ceph-mon[30237]: from='osd.7 v1:172.21.15.181:6812/1269032985' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:09:36.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:36 smithi181 ceph-mon[30237]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:09:36.301 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 7 on host 'smithi181' 2024-04-27T01:09:36.302 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:36.311+0000 7f4f22ffd700 1 -- 172.21.15.181:0/199435298 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (unknown 0 0 3858794168) 0x7f4f10000bc0 con 0x7f4f18071060 2024-04-27T01:09:36.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:36.313+0000 7f4f33b9a700 1 -- 172.21.15.181:0/199435298 >> v1:172.21.15.162:6800/2166195007 conn(0x7f4f18071060 legacy=0x7f4f18073520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:36.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:36.313+0000 7f4f33b9a700 1 -- 172.21.15.181:0/199435298 >> v1:172.21.15.181:6789/0 conn(0x7f4f2c1053f0 legacy=0x7f4f2c19cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:36.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:36.313+0000 7f4f33b9a700 1 -- 172.21.15.181:0/199435298 shutdown_connections 2024-04-27T01:09:36.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:36.313+0000 7f4f33b9a700 1 -- 172.21.15.181:0/199435298 >> 172.21.15.181:0/199435298 conn(0x7f4f2c0fbd00 msgr2=0x7f4f2c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:36.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:36.313+0000 7f4f33b9a700 1 -- 172.21.15.181:0/199435298 shutdown_connections 2024-04-27T01:09:36.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:36.313+0000 7f4f33b9a700 1 -- 172.21.15.181:0/199435298 wait complete. 2024-04-27T01:09:36.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:36 smithi162 ceph-mon[26714]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:36.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:36 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:36.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:36 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:36.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:36 smithi162 ceph-mon[26714]: from='osd.7 v1:172.21.15.181:6812/1269032985' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:09:36.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:36 smithi162 ceph-mon[26714]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:09:36.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:36 smithi162 ceph-mon[37680]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:36.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:36 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:36.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:36 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:36.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:36 smithi162 ceph-mon[37680]: from='osd.7 v1:172.21.15.181:6812/1269032985' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:09:36.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:36 smithi162 ceph-mon[37680]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-27T01:09:37.085 DEBUG:teuthology.orchestra.run.smithi181:osd.7> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.7.service 2024-04-27T01:09:37.087 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-27T01:09:37.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-27T01:09:37.341 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:09:37.359 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:37.359 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:37.359 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:37 smithi181 ceph-mon[30237]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T01:09:37.359 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:37 smithi181 ceph-mon[30237]: from='osd.7 v1:172.21.15.181:6812/1269032985' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:37.360 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:37 smithi181 ceph-mon[30237]: osdmap e53: 8 total, 7 up, 8 in 2024-04-27T01:09:37.360 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:37 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:37.360 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:37 smithi181 ceph-mon[30237]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:37.360 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:37 smithi181 ceph-mon[30237]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:37.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:37.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:37.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[37680]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T01:09:37.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[37680]: from='osd.7 v1:172.21.15.181:6812/1269032985' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:37.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[37680]: osdmap e53: 8 total, 7 up, 8 in 2024-04-27T01:09:37.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:37.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[37680]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:37.454 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[37680]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:37.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:37.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:37.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[26714]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-27T01:09:37.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[26714]: from='osd.7 v1:172.21.15.181:6812/1269032985' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:37.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[26714]: osdmap e53: 8 total, 7 up, 8 in 2024-04-27T01:09:37.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:37.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[26714]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2024-04-27T01:09:37.455 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:37 smithi162 ceph-mon[26714]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:38.262 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:09:37 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:09:37.943+0000 7fc11b27b700 -1 osd.7 0 waiting for initial osdmap 2024-04-27T01:09:38.262 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:09:37 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:09:37.949+0000 7fc113066700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T01:09:38.590 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:38 smithi181 ceph-mon[30237]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:09:38.590 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:38 smithi181 ceph-mon[30237]: osdmap e54: 8 total, 7 up, 8 in 2024-04-27T01:09:38.591 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:38 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:38.591 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:38 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:38.591 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:38 smithi181 ceph-mon[30237]: from='osd.7 ' entity='osd.7' 2024-04-27T01:09:38.742 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:38 smithi162 ceph-mon[26714]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:09:38.742 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:38 smithi162 ceph-mon[26714]: osdmap e54: 8 total, 7 up, 8 in 2024-04-27T01:09:38.742 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:38 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:38.742 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:38 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:38.742 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:38 smithi162 ceph-mon[26714]: from='osd.7 ' entity='osd.7' 2024-04-27T01:09:38.742 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:38 smithi162 ceph-mon[37680]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2024-04-27T01:09:38.742 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:38 smithi162 ceph-mon[37680]: osdmap e54: 8 total, 7 up, 8 in 2024-04-27T01:09:38.743 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:38 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:38.743 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:38 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:38.743 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:38 smithi162 ceph-mon[37680]: from='osd.7 ' entity='osd.7' 2024-04-27T01:09:38.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.968+0000 7f6b49239700 1 -- 172.21.15.162:0/260994763 >> v1:172.21.15.162:6790/0 conn(0x7f6b441053b0 legacy=0x7f6b441077a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:38.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.968+0000 7f6b49239700 1 -- 172.21.15.162:0/260994763 shutdown_connections 2024-04-27T01:09:38.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.968+0000 7f6b49239700 1 -- 172.21.15.162:0/260994763 >> 172.21.15.162:0/260994763 conn(0x7f6b440fc500 msgr2=0x7f6b440fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:38.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.968+0000 7f6b49239700 1 -- 172.21.15.162:0/260994763 shutdown_connections 2024-04-27T01:09:38.960 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.968+0000 7f6b49239700 1 -- 172.21.15.162:0/260994763 wait complete. 2024-04-27T01:09:38.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b49239700 1 Processor -- start 2024-04-27T01:09:38.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b49239700 1 -- start start 2024-04-27T01:09:38.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b49239700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b4406aaa0 con 0x7f6b44100f10 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b49239700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b441a3730 con 0x7f6b441053b0 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b49239700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f6b441a4950 con 0x7f6b44109040 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b42d9d700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6b44100f10 0x7f6b4419a1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:41564/0 (socket says 172.21.15.162:41564) 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b42d9d700 1 -- 172.21.15.162:0/2598035872 learned_addr learned my addr 172.21.15.162:0/2598035872 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 594946933 0 0) 0x7f6b441a4950 con 0x7f6b44109040 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b2c005240 con 0x7f6b44109040 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762292628 0 0) 0x7f6b4406aaa0 con 0x7f6b44100f10 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b441a4950 con 0x7f6b44100f10 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2977845732 0 0) 0x7f6b441a3730 con 0x7f6b441053b0 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.969+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b4406aaa0 con 0x7f6b441053b0 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1194799216 0 0) 0x7f6b4406aaa0 con 0x7f6b441053b0 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b441a3730 con 0x7f6b441053b0 2024-04-27T01:09:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6b34001860 con 0x7f6b441053b0 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2945899544 0 0) 0x7f6b2c005240 con 0x7f6b44109040 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b4406aaa0 con 0x7f6b44109040 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1158797103 0 0) 0x7f6b441a4950 con 0x7f6b44100f10 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6b2c005240 con 0x7f6b44100f10 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6b3c001540 con 0x7f6b44109040 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6b380017f0 con 0x7f6b44100f10 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3432823114 0 0) 0x7f6b441a3730 con 0x7f6b441053b0 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 >> v1:172.21.15.162:6790/0 conn(0x7f6b44109040 legacy=0x7f6b441a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 >> v1:172.21.15.162:6789/0 conn(0x7f6b44100f10 legacy=0x7f6b4419a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b441a5b90 con 0x7f6b441053b0 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b49239700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6b441a4b80 con 0x7f6b441053b0 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b49239700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6b441a50c0 con 0x7f6b441053b0 2024-04-27T01:09:38.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f6b34004660 con 0x7f6b441053b0 2024-04-27T01:09:38.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.970+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6b34004d50 con 0x7f6b441053b0 2024-04-27T01:09:38.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.971+0000 7f6b49239700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b30005150 con 0x7f6b441053b0 2024-04-27T01:09:38.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.972+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f6b34005040 con 0x7f6b441053b0 2024-04-27T01:09:38.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.972+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(56..56 src has 1..56) ==== 4110+0+0 (unknown 3217284677 0 0) 0x7f6b34087b20 con 0x7f6b441053b0 2024-04-27T01:09:38.968 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:38.977+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f6b34004d50 con 0x7f6b441053b0 2024-04-27T01:09:39.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:39.183+0000 7f6b49239700 1 -- 172.21.15.162:0/2598035872 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f6b30005440 con 0x7f6b441053b0 2024-04-27T01:09:39.174 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:39.183+0000 7f6b2bfff700 1 -- 172.21.15.162:0/2598035872 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v56) ==== 74+0+130 (unknown 401554801 0 737352282) 0x7f6b34055a90 con 0x7f6b441053b0 2024-04-27T01:09:39.175 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:09:39.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:39.186+0000 7f6b49239700 1 -- 172.21.15.162:0/2598035872 >> v1:172.21.15.162:6800/2166195007 conn(0x7f6b2c06eee0 legacy=0x7f6b2c0713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:39.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:39.186+0000 7f6b49239700 1 -- 172.21.15.162:0/2598035872 >> v1:172.21.15.181:6789/0 conn(0x7f6b441053b0 legacy=0x7f6b4419d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:39.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:39.186+0000 7f6b49239700 1 -- 172.21.15.162:0/2598035872 shutdown_connections 2024-04-27T01:09:39.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:39.186+0000 7f6b49239700 1 -- 172.21.15.162:0/2598035872 >> 172.21.15.162:0/2598035872 conn(0x7f6b440fc500 msgr2=0x7f6b4410b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:39.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:39.186+0000 7f6b49239700 1 -- 172.21.15.162:0/2598035872 shutdown_connections 2024-04-27T01:09:39.177 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:39.186+0000 7f6b49239700 1 -- 172.21.15.162:0/2598035872 wait complete. 2024-04-27T01:09:39.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[37680]: purged_snaps scrub starts 2024-04-27T01:09:39.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[37680]: purged_snaps scrub ok 2024-04-27T01:09:39.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[37680]: osd.7 v1:172.21.15.181:6812/1269032985 boot 2024-04-27T01:09:39.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[37680]: osdmap e55: 8 total, 8 up, 8 in 2024-04-27T01:09:39.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:39.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[37680]: osdmap e56: 8 total, 8 up, 8 in 2024-04-27T01:09:39.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[37680]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:39.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2598035872' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T01:09:39.819 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[26714]: purged_snaps scrub starts 2024-04-27T01:09:39.819 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[26714]: purged_snaps scrub ok 2024-04-27T01:09:39.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[26714]: osd.7 v1:172.21.15.181:6812/1269032985 boot 2024-04-27T01:09:39.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[26714]: osdmap e55: 8 total, 8 up, 8 in 2024-04-27T01:09:39.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:39.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[26714]: osdmap e56: 8 total, 8 up, 8 in 2024-04-27T01:09:39.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[26714]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:39.820 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2598035872' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T01:09:39.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:39 smithi181 ceph-mon[30237]: purged_snaps scrub starts 2024-04-27T01:09:39.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:39 smithi181 ceph-mon[30237]: purged_snaps scrub ok 2024-04-27T01:09:39.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:39 smithi181 ceph-mon[30237]: osd.7 v1:172.21.15.181:6812/1269032985 boot 2024-04-27T01:09:39.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:39 smithi181 ceph-mon[30237]: osdmap e55: 8 total, 8 up, 8 in 2024-04-27T01:09:39.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:39 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:09:39.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:39 smithi181 ceph-mon[30237]: osdmap e56: 8 total, 8 up, 8 in 2024-04-27T01:09:39.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:39 smithi181 ceph-mon[30237]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-04-27T01:09:39.845 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2598035872' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-27T01:09:39.846 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1714180178,"num_in_osds":8,"osd_in_since":1714180160,"num_remapped_pgs":1} 2024-04-27T01:09:39.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T01:09:40.112 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: osdmap e57: 8 total, 8 up, 8 in 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: Detected new or changed devices on smithi181 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi181 to 4272M 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:41.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:41.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:41.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:41.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:41.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:40 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: osdmap e57: 8 total, 8 up, 8 in 2024-04-27T01:09:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: Detected new or changed devices on smithi181 2024-04-27T01:09:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi181 to 4272M 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: osdmap e57: 8 total, 8 up, 8 in 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: Detected new or changed devices on smithi181 2024-04-27T01:09:41.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi181 to 4272M 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:41.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:41.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:40 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:41.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.648+0000 7eff7e549700 1 -- 172.21.15.162:0/742515350 >> v1:172.21.15.162:6789/0 conn(0x7eff7810d9b0 legacy=0x7eff7810fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:41.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.648+0000 7eff7e549700 1 -- 172.21.15.162:0/742515350 shutdown_connections 2024-04-27T01:09:41.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.648+0000 7eff7e549700 1 -- 172.21.15.162:0/742515350 >> 172.21.15.162:0/742515350 conn(0x7eff78076220 msgr2=0x7eff78076630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:41.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.648+0000 7eff7e549700 1 -- 172.21.15.162:0/742515350 shutdown_connections 2024-04-27T01:09:41.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.648+0000 7eff7e549700 1 -- 172.21.15.162:0/742515350 wait complete. 2024-04-27T01:09:41.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.649+0000 7eff7e549700 1 Processor -- start 2024-04-27T01:09:41.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.649+0000 7eff7e549700 1 -- start start 2024-04-27T01:09:41.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff7e549700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff781a70d0 con 0x7eff78109e10 2024-04-27T01:09:41.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff7e549700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff781a82f0 con 0x7eff7810d9b0 2024-04-27T01:09:41.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff7e549700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7eff781a9530 con 0x7eff78101b70 2024-04-27T01:09:41.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff77fff700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7eff78101b70 0x7eff78101150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:50512/0 (socket says 172.21.15.162:50512) 2024-04-27T01:09:41.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff77fff700 1 -- 172.21.15.162:0/1059742930 learned_addr learned my addr 172.21.15.162:0/1059742930 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:09:41.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff777fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7eff78109e10 0x7eff781a1e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:41578/0 (socket says 172.21.15.162:41578) 2024-04-27T01:09:41.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247152069 0 0) 0x7eff781a9530 con 0x7eff78101b70 2024-04-27T01:09:41.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff600035b0 con 0x7eff78101b70 2024-04-27T01:09:41.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1945585256 0 0) 0x7eff781a70d0 con 0x7eff78109e10 2024-04-27T01:09:41.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff781a9530 con 0x7eff78109e10 2024-04-27T01:09:41.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 974279720 0 0) 0x7eff781a82f0 con 0x7eff7810d9b0 2024-04-27T01:09:41.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff781a70d0 con 0x7eff7810d9b0 2024-04-27T01:09:41.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2497062535 0 0) 0x7eff781a9530 con 0x7eff78109e10 2024-04-27T01:09:41.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff781a82f0 con 0x7eff78109e10 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7eff6c0018b0 con 0x7eff78109e10 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1164203454 0 0) 0x7eff600035b0 con 0x7eff78101b70 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff781a9530 con 0x7eff78101b70 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7eff680012d0 con 0x7eff78101b70 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 134150350 0 0) 0x7eff781a70d0 con 0x7eff7810d9b0 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff600035b0 con 0x7eff7810d9b0 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7eff70001bc0 con 0x7eff7810d9b0 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3571804696 0 0) 0x7eff781a82f0 con 0x7eff78109e10 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 >> v1:172.21.15.162:6790/0 conn(0x7eff78101b70 legacy=0x7eff78101150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 >> v1:172.21.15.181:6789/0 conn(0x7eff7810d9b0 legacy=0x7eff781a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.650+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff781aa770 con 0x7eff78109e10 2024-04-27T01:09:41.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.651+0000 7eff7e549700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff781a7300 con 0x7eff78109e10 2024-04-27T01:09:41.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.651+0000 7eff7e549700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7eff781a77c0 con 0x7eff78109e10 2024-04-27T01:09:41.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.651+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7eff6c001c60 con 0x7eff78109e10 2024-04-27T01:09:41.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.651+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7eff6c005d10 con 0x7eff78109e10 2024-04-27T01:09:41.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.652+0000 7eff7e549700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff64005120 con 0x7eff78109e10 2024-04-27T01:09:41.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.652+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7eff6c01c730 con 0x7eff78109e10 2024-04-27T01:09:41.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.653+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(58..58 src has 1..58) ==== 4081+0+0 (unknown 1707219791 0 0) 0x7eff6c088c60 con 0x7eff78109e10 2024-04-27T01:09:41.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.657+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7eff6c056d20 con 0x7eff78109e10 2024-04-27T01:09:41.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.856+0000 7eff7e549700 1 -- 172.21.15.162:0/1059742930 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff64005410 con 0x7eff78109e10 2024-04-27T01:09:41.849 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.857+0000 7eff757fa700 1 -- 172.21.15.162:0/1059742930 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) ==== 74+0+11525 (unknown 1651271748 0 3666711925) 0x7eff6c056620 con 0x7eff78109e10 2024-04-27T01:09:41.849 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:09:41.849 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":59,"fsid":"ea378f3c-0431-11ef-bc93-c7b262605968","created":"2024-04-27T01:04:19.059661+0000","modified":"2024-04-27T01:09:41.768367+0000","last_up_change":"2024-04-27T01:09:38.516534+0000","last_in_change":"2024-04-27T01:09:20.973588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T01:07:37.280446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8f1e299d-4d29-4333-b632-249ff9f12d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":2041814805}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":2041814805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":2041814805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":2041814805}]},"public_addr":"172.21.15.162:6801/2041814805","cluster_addr":"172.21.15.162:6802/2041814805","heartbeat_back_addr":"172.21.15.162:6804/2041814805","heartbeat_front_addr":"172.21.15.162:6803/2041814805","state":["exists","up"]},{"osd":1,"uuid":"9013bcef-adce-49f2-952c-f3f208bf4e96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":680367434}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":680367434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":680367434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":680367434}]},"public_addr":"172.21.15.162:6805/680367434","cluster_addr":"172.21.15.162:6806/680367434","heartbeat_back_addr":"172.21.15.162:6808/680367434","heartbeat_front_addr":"172.21.15.162:6807/680367434","state":["exists","up"]},{"osd":2,"uuid":"f2212181-797c-4366-b087-2522d5db9e95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":2147722624}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":2147722624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":2147722624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":2147722624}]},"public_addr":"172.21.15.162:6809/2147722624","cluster_addr":"172.21.15.162:6810/2147722624","heartbeat_back_addr":"172.21.15.162:6812/2147722624","heartbeat_front_addr":"172.21.15.162:6811/2147722624","state":["exists","up"]},{"osd":3,"uuid":"21e98e27-d849-4b8c-9db9-98e738395fd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":2875926639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":2875926639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6816","nonce":2875926639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":2875926639}]},"public_addr":"172.21.15.162:6813/2875926639","cluster_addr":"172.21.15.162:6814/2875926639","heartbeat_back_addr":"172.21.15.162:6816/2875926639","heartbeat_front_addr":"172.21.15.162:6815/2875926639","state":["exists","up"]},{"osd":4,"uuid":"ad7cfc95-49e8-440a-8e15-55e6900ae122","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6800","nonce":1869417285}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6801","nonce":1869417285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6803","nonce":1869417285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6802","nonce":1869417285}]},"public_addr":"172.21.15.181:6800/1869417285","cluster_addr":"172.21.15.181:6801/1869417285","heartbeat_back_addr":"172.21.15.181:6803/1869417285","heartbeat_front_addr":"172.21.15.181:6802/1869417285","state":["exists","up"]},{"osd":5,"uuid":"46341ac9-3fc7-49c0-bc92-03ba76da4ec7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6804","nonce":3172409639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6805","nonce":3172409639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6807","nonce":3172409639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6806","nonce":3172409639}]},"public_addr":"172.21.15.181:6804/3172409639","cluster_addr":"172.21.15.181:6805/3172409639","heartbeat_back_addr":"172.21.15.181:6807/3172409639","heartbeat_front_addr":"172.21.15.181:6806/3172409639","state":["exists","up"]},{"osd":6,"uuid":"832670c0-ca6f-484e-8719-3cf680f03fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6808","nonce":30247032}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6809","nonce":30247032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6811","nonce":30247032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6810","nonce":30247032}]},"public_addr":"172.21.15.181:6808/30247032","cluster_addr":"172.21.15.181:6809/30247032","heartbeat_back_addr":"172.21.15.181:6811/30247032","heartbeat_front_addr":"172.21.15.181:6810/30247032","state":["exists","up"]},{"osd":7,"uuid":"63a81b98-465c-4a37-a666-90133708abc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6812","nonce":1269032985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6813","nonce":1269032985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6815","nonce":1269032985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6814","nonce":1269032985}]},"public_addr":"172.21.15.181:6812/1269032985","cluster_addr":"172.21.15.181:6813/1269032985","heartbeat_back_addr":"172.21.15.181:6815/1269032985","heartbeat_front_addr":"172.21.15.181:6814/1269032985","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:06:41.989409+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:07:06.920751+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:07:32.636741+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:07:58.313419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:08:21.343521+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:08:46.671407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:09:12.035903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:09:36.866889+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:0/165265347":"2024-04-28T01:05:20.958366+0000","172.21.15.162:0/1636118121":"2024-04-28T01:04:54.155544+0000","172.21.15.162:0/837853490":"2024-04-28T01:04:54.155544+0000","172.21.15.162:0/3941236233":"2024-04-28T01:05:20.958366+0000","172.21.15.162:0/104247678":"2024-04-28T01:04:54.155544+0000","172.21.15.162:0/2952006630":"2024-04-28T01:05:20.958366+0000","172.21.15.162:6800/629135069":"2024-04-28T01:04:54.155544+0000","172.21.15.162:6800/106205076":"2024-04-28T01:05:20.958366+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T01:09:41.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.859+0000 7eff7e549700 1 -- 172.21.15.162:0/1059742930 >> v1:172.21.15.162:6800/2166195007 conn(0x7eff6006ede0 legacy=0x7eff600712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:41.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.859+0000 7eff7e549700 1 -- 172.21.15.162:0/1059742930 >> v1:172.21.15.162:6789/0 conn(0x7eff78109e10 legacy=0x7eff781a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:41.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.859+0000 7eff7e549700 1 -- 172.21.15.162:0/1059742930 shutdown_connections 2024-04-27T01:09:41.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.859+0000 7eff7e549700 1 -- 172.21.15.162:0/1059742930 >> 172.21.15.162:0/1059742930 conn(0x7eff78076220 msgr2=0x7eff781111c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:41.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.859+0000 7eff7e549700 1 -- 172.21.15.162:0/1059742930 shutdown_connections 2024-04-27T01:09:41.851 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:41.859+0000 7eff7e549700 1 -- 172.21.15.162:0/1059742930 wait complete. 2024-04-27T01:09:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:41 smithi162 ceph-mon[26714]: osdmap e58: 8 total, 8 up, 8 in 2024-04-27T01:09:42.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:41 smithi162 ceph-mon[26714]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:09:42.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:41 smithi162 ceph-mon[37680]: osdmap e58: 8 total, 8 up, 8 in 2024-04-27T01:09:42.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:41 smithi162 ceph-mon[37680]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:09:42.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:41 smithi181 ceph-mon[30237]: osdmap e58: 8 total, 8 up, 8 in 2024-04-27T01:09:42.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:41 smithi181 ceph-mon[30237]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:09:42.494 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-27T01:07:37.280446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-27T01:09:42.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-27T01:09:42.734 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:09:42.764 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:42 smithi162 ceph-mon[37680]: osdmap e59: 8 total, 8 up, 8 in 2024-04-27T01:09:42.764 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1059742930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:09:43.023 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:42 smithi162 ceph-mon[26714]: osdmap e59: 8 total, 8 up, 8 in 2024-04-27T01:09:43.023 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1059742930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:09:43.215 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:42 smithi181 ceph-mon[30237]: osdmap e59: 8 total, 8 up, 8 in 2024-04-27T01:09:43.215 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1059742930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:09:43.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:43 smithi162 ceph-mon[26714]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:09:43.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:43 smithi162 ceph-mon[37680]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:09:44.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.198+0000 7f671745e700 1 -- 172.21.15.162:0/1657889436 >> v1:172.21.15.162:6789/0 conn(0x7f6710102250 legacy=0x7f6710102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:44.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.198+0000 7f671745e700 1 -- 172.21.15.162:0/1657889436 shutdown_connections 2024-04-27T01:09:44.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.198+0000 7f671745e700 1 -- 172.21.15.162:0/1657889436 >> 172.21.15.162:0/1657889436 conn(0x7f67100fdb20 msgr2=0x7f67100fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:44.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.198+0000 7f671745e700 1 -- 172.21.15.162:0/1657889436 shutdown_connections 2024-04-27T01:09:44.190 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.198+0000 7f671745e700 1 -- 172.21.15.162:0/1657889436 wait complete. 2024-04-27T01:09:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.199+0000 7f671745e700 1 Processor -- start 2024-04-27T01:09:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.199+0000 7f671745e700 1 -- start start 2024-04-27T01:09:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.199+0000 7f671745e700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67101a6e70 con 0x7f671010aee0 2024-04-27T01:09:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.199+0000 7f671745e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67101a8090 con 0x7f6710102250 2024-04-27T01:09:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.199+0000 7f671745e700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f67101a92d0 con 0x7f671010ea80 2024-04-27T01:09:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.199+0000 7f67149f9700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f671010aee0 0x7f67101a1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:41598/0 (socket says 172.21.15.162:41598) 2024-04-27T01:09:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.199+0000 7f67159fb700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f671010ea80 0x7f67101a5620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:50530/0 (socket says 172.21.15.162:50530) 2024-04-27T01:09:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.199+0000 7f67149f9700 1 -- 172.21.15.162:0/350651155 learned_addr learned my addr 172.21.15.162:0/350651155 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:09:44.193 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127770519 0 0) 0x7f67101a6e70 con 0x7f671010aee0 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66fc0031f0 con 0x7f671010aee0 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4025533423 0 0) 0x7f67101a92d0 con 0x7f671010ea80 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67101a6e70 con 0x7f671010ea80 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 910647668 0 0) 0x7f67101a8090 con 0x7f6710102250 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67101a92d0 con 0x7f6710102250 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 960385654 0 0) 0x7f66fc0031f0 con 0x7f671010aee0 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67101a8090 con 0x7f671010aee0 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f670c001a30 con 0x7f671010aee0 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3618796569 0 0) 0x7f67101a6e70 con 0x7f671010ea80 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f66fc0031f0 con 0x7f671010ea80 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6708001b50 con 0x7f671010ea80 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2745505263 0 0) 0x7f67101a92d0 con 0x7f6710102250 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67101a6e70 con 0x7f6710102250 2024-04-27T01:09:44.194 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f67000044f0 con 0x7f6710102250 2024-04-27T01:09:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3870617815 0 0) 0x7f67101a8090 con 0x7f671010aee0 2024-04-27T01:09:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 >> v1:172.21.15.162:6790/0 conn(0x7f671010ea80 legacy=0x7f67101a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 >> v1:172.21.15.181:6789/0 conn(0x7f6710102250 legacy=0x7f6710107310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67101aa510 con 0x7f671010aee0 2024-04-27T01:09:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f671745e700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f67101a94a0 con 0x7f671010aee0 2024-04-27T01:09:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f671745e700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f67101a9960 con 0x7f671010aee0 2024-04-27T01:09:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.200+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f670c001ce0 con 0x7f671010aee0 2024-04-27T01:09:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.201+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f670c005c00 con 0x7f671010aee0 2024-04-27T01:09:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.201+0000 7f671745e700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66f4005120 con 0x7f671010aee0 2024-04-27T01:09:44.195 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.202+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f670c003930 con 0x7f671010aee0 2024-04-27T01:09:44.196 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.202+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(59..59 src has 1..59) ==== 4081+0+0 (unknown 102955441 0 0) 0x7f670c088b20 con 0x7f671010aee0 2024-04-27T01:09:44.198 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.207+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f670c005c00 con 0x7f671010aee0 2024-04-27T01:09:44.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:43 smithi181 ceph-mon[30237]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:09:44.393 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.402+0000 7f671745e700 1 -- 172.21.15.162:0/350651155 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f66f4005c20 con 0x7f671010aee0 2024-04-27T01:09:44.394 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.402+0000 7f67067fc700 1 -- 172.21.15.162:0/350651155 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) ==== 93+0+10 (unknown 4013917427 0 2170607528) 0x7f670c056aa0 con 0x7f671010aee0 2024-04-27T01:09:44.394 INFO:teuthology.orchestra.run.smithi162.stdout:pg_num: 1 2024-04-27T01:09:44.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.404+0000 7f671745e700 1 -- 172.21.15.162:0/350651155 >> v1:172.21.15.162:6800/2166195007 conn(0x7f66fc06eef0 legacy=0x7f66fc0713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:44.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.404+0000 7f671745e700 1 -- 172.21.15.162:0/350651155 >> v1:172.21.15.162:6789/0 conn(0x7f671010aee0 legacy=0x7f67101a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:44.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.404+0000 7f671745e700 1 -- 172.21.15.162:0/350651155 shutdown_connections 2024-04-27T01:09:44.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.404+0000 7f671745e700 1 -- 172.21.15.162:0/350651155 >> 172.21.15.162:0/350651155 conn(0x7f67100fdb20 msgr2=0x7f671010ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:44.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.404+0000 7f671745e700 1 -- 172.21.15.162:0/350651155 shutdown_connections 2024-04-27T01:09:44.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:09:44.405+0000 7f671745e700 1 -- 172.21.15.162:0/350651155 wait complete. 2024-04-27T01:09:45.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/350651155' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T01:09:45.026 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/350651155' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T01:09:45.055 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi162 2024-04-27T01:09:45.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi162=foo.a' 2024-04-27T01:09:45.082 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/350651155' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T01:09:45.304 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:09:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:45 smithi181 ceph-mon[30237]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-27T01:09:46.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:45 smithi162 ceph-mon[26714]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-27T01:09:46.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:45 smithi162 ceph-mon[37680]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-27T01:09:46.720 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.727+0000 7f22db5f8700 1 -- 172.21.15.181:0/1104288177 >> v1:172.21.15.181:6789/0 conn(0x7f22d41065f0 legacy=0x7f22d41089e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:46.720 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.727+0000 7f22db5f8700 1 -- 172.21.15.181:0/1104288177 shutdown_connections 2024-04-27T01:09:46.720 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.727+0000 7f22db5f8700 1 -- 172.21.15.181:0/1104288177 >> 172.21.15.181:0/1104288177 conn(0x7f22d40fbd00 msgr2=0x7f22d40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:46.720 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.727+0000 7f22db5f8700 1 -- 172.21.15.181:0/1104288177 shutdown_connections 2024-04-27T01:09:46.720 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.727+0000 7f22db5f8700 1 -- 172.21.15.181:0/1104288177 wait complete. 2024-04-27T01:09:46.720 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.727+0000 7f22db5f8700 1 Processor -- start 2024-04-27T01:09:46.720 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.727+0000 7f22db5f8700 1 -- start start 2024-04-27T01:09:46.720 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22db5f8700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22d41986c0 con 0x7f22d4100390 2024-04-27T01:09:46.720 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22db5f8700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22d41a3eb0 con 0x7f22d41065f0 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22db5f8700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f22d41a5090 con 0x7f22d410a220 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22d8b93700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f22d41065f0 0x7f22d419e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:55794/0 (socket says 172.21.15.181:55794) 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22d8b93700 1 -- 172.21.15.181:0/2720948198 learned_addr learned my addr 172.21.15.181:0/2720948198 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1447839267 0 0) 0x7f22d41a3eb0 con 0x7f22d41065f0 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22c00035b0 con 0x7f22d41065f0 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399218045 0 0) 0x7f22d41986c0 con 0x7f22d4100390 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d41a3eb0 con 0x7f22d4100390 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3455824880 0 0) 0x7f22d41a5090 con 0x7f22d410a220 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d41986c0 con 0x7f22d410a220 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 310344706 0 0) 0x7f22c00035b0 con 0x7f22d41065f0 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22d41a5090 con 0x7f22d41065f0 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f22c4001430 con 0x7f22d41065f0 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 129315404 0 0) 0x7f22d41986c0 con 0x7f22d410a220 2024-04-27T01:09:46.721 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22c00035b0 con 0x7f22d410a220 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f22cc001af0 con 0x7f22d410a220 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1678980575 0 0) 0x7f22d41a3eb0 con 0x7f22d4100390 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22d41986c0 con 0x7f22d4100390 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3113006050 0 0) 0x7f22d41a5090 con 0x7f22d41065f0 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.728+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 >> v1:172.21.15.162:6790/0 conn(0x7f22d410a220 legacy=0x7f22d41a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.729+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 >> v1:172.21.15.162:6789/0 conn(0x7f22d4100390 legacy=0x7f22d4197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.729+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d41a62b0 con 0x7f22d41065f0 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.729+0000 7f22db5f8700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22d41a2f00 con 0x7f22d41065f0 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.729+0000 7f22db5f8700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f22d41a3440 con 0x7f22d41065f0 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.729+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f22c4003740 con 0x7f22d41065f0 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.729+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f22c4004f30 con 0x7f22d41065f0 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.729+0000 7f22db5f8700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b8005150 con 0x7f22d41065f0 2024-04-27T01:09:46.722 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.730+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f22c4003ab0 con 0x7f22d41065f0 2024-04-27T01:09:46.723 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.731+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(59..59 src has 1..59) ==== 4081+0+0 (unknown 102955441 0 0) 0x7f22c4088c50 con 0x7f22d41065f0 2024-04-27T01:09:46.725 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.735+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f22c4056c30 con 0x7f22d41065f0 2024-04-27T01:09:46.931 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.941+0000 7f22db5f8700 1 -- 172.21.15.181:0/2720948198 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi162=foo.a", "target": ["mon-mgr", ""]}) -- 0x7f22b8000bc0 con 0x7f22c006ede0 2024-04-27T01:09:46.938 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.948+0000 7f22ca7fc700 1 -- 172.21.15.181:0/2720948198 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (unknown 0 0 1123153589) 0x7f22b8000bc0 con 0x7f22c006ede0 2024-04-27T01:09:46.938 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled rgw.foo.a update... 2024-04-27T01:09:46.940 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.950+0000 7f22db5f8700 1 -- 172.21.15.181:0/2720948198 >> v1:172.21.15.162:6800/2166195007 conn(0x7f22c006ede0 legacy=0x7f22c00712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:46.940 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.950+0000 7f22db5f8700 1 -- 172.21.15.181:0/2720948198 >> v1:172.21.15.181:6789/0 conn(0x7f22d41065f0 legacy=0x7f22d419e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:46.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.950+0000 7f22db5f8700 1 -- 172.21.15.181:0/2720948198 shutdown_connections 2024-04-27T01:09:46.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.950+0000 7f22db5f8700 1 -- 172.21.15.181:0/2720948198 >> 172.21.15.181:0/2720948198 conn(0x7f22d40fbd00 msgr2=0x7f22d40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:46.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.950+0000 7f22db5f8700 1 -- 172.21.15.181:0/2720948198 shutdown_connections 2024-04-27T01:09:46.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:46.950+0000 7f22db5f8700 1 -- 172.21.15.181:0/2720948198 wait complete. 2024-04-27T01:09:47.500 DEBUG:teuthology.orchestra.run.smithi162:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@rgw.foo.a.service 2024-04-27T01:09:47.503 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi181 2024-04-27T01:09:47.503 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-04-27T01:09:47.778 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:09:48.217 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: from='client.24358 v1:172.21.15.181:0/2720948198' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi162=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:48.218 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-04-27T01:09:48.218 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:48.218 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:48.218 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-27T01:09:48.218 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:48.218 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:48.218 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:48.218 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:09:48.218 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T01:09:48.218 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:48.219 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:47 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: from='client.24358 v1:172.21.15.181:0/2720948198' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi162=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:48.259 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: from='client.24358 v1:172.21.15.181:0/2720948198' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi162=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:48.260 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:47 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:49.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:48 smithi162 ceph-mon[37680]: Deploying daemon rgw.foo.a on smithi162 2024-04-27T01:09:49.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:48 smithi162 ceph-mon[26714]: Deploying daemon rgw.foo.a on smithi162 2024-04-27T01:09:49.114 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:48 smithi181 ceph-mon[30237]: Deploying daemon rgw.foo.a on smithi162 2024-04-27T01:09:49.304 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.314+0000 7f79b1360700 1 -- 172.21.15.181:0/1270823588 >> v1:172.21.15.181:6789/0 conn(0x7f79ac1064a0 legacy=0x7f79ac1088f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:49.304 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.314+0000 7f79b1360700 1 -- 172.21.15.181:0/1270823588 shutdown_connections 2024-04-27T01:09:49.304 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.314+0000 7f79b1360700 1 -- 172.21.15.181:0/1270823588 >> 172.21.15.181:0/1270823588 conn(0x7f79ac0fd630 msgr2=0x7f79ac0ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:49.304 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.314+0000 7f79b1360700 1 -- 172.21.15.181:0/1270823588 shutdown_connections 2024-04-27T01:09:49.304 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.314+0000 7f79b1360700 1 -- 172.21.15.181:0/1270823588 wait complete. 2024-04-27T01:09:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.314+0000 7f79b1360700 1 Processor -- start 2024-04-27T01:09:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79b1360700 1 -- start start 2024-04-27T01:09:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79b1360700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79ac1a4880 con 0x7f79ac10a1e0 2024-04-27T01:09:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79b1360700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79ac1a5ac0 con 0x7f79ac1064a0 2024-04-27T01:09:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79b1360700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f79ac1a6d00 con 0x7f79ac102070 2024-04-27T01:09:49.306 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79a27fc700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f79ac1064a0 0x7f79ac19f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:55806/0 (socket says 172.21.15.181:55806) 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79a27fc700 1 -- 172.21.15.181:0/1950307810 learned_addr learned my addr 172.21.15.181:0/1950307810 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 520543181 0 0) 0x7f79ac1a5ac0 con 0x7f79ac1064a0 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79980035b0 con 0x7f79ac1064a0 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4059869821 0 0) 0x7f79ac1a4880 con 0x7f79ac10a1e0 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79ac1a5ac0 con 0x7f79ac10a1e0 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4054440459 0 0) 0x7f79980035b0 con 0x7f79ac1064a0 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79ac1a4880 con 0x7f79ac1064a0 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.315+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f79940019d0 con 0x7f79ac1064a0 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.316+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2865766369 0 0) 0x7f79ac1a6d00 con 0x7f79ac102070 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.316+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79980035b0 con 0x7f79ac102070 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.316+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 950756074 0 0) 0x7f79ac1a4880 con 0x7f79ac1064a0 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.316+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 >> v1:172.21.15.162:6790/0 conn(0x7f79ac102070 legacy=0x7f79ac075ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.316+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 >> v1:172.21.15.162:6789/0 conn(0x7f79ac10a1e0 legacy=0x7f79ac1a2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:49.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.316+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac1a7f40 con 0x7f79ac1064a0 2024-04-27T01:09:49.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.316+0000 7f79b1360700 1 -- 172.21.15.181:0/1950307810 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f79ac1a6f30 con 0x7f79ac1064a0 2024-04-27T01:09:49.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.316+0000 7f79b1360700 1 -- 172.21.15.181:0/1950307810 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f79ac1a7470 con 0x7f79ac1064a0 2024-04-27T01:09:49.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.316+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f79940047f0 con 0x7f79ac1064a0 2024-04-27T01:09:49.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.316+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7994004f50 con 0x7f79ac1064a0 2024-04-27T01:09:49.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.317+0000 7f79b1360700 1 -- 172.21.15.181:0/1950307810 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f798c005150 con 0x7f79ac1064a0 2024-04-27T01:09:49.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.317+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f79940037f0 con 0x7f79ac1064a0 2024-04-27T01:09:49.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.318+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(59..59 src has 1..59) ==== 4081+0+0 (unknown 102955441 0 0) 0x7f7994088e70 con 0x7f79ac1064a0 2024-04-27T01:09:49.312 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.322+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f7994056f30 con 0x7f79ac1064a0 2024-04-27T01:09:49.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.520+0000 7f79b1360700 1 -- 172.21.15.181:0/1950307810 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) -- 0x7f798c005440 con 0x7f79ac1064a0 2024-04-27T01:09:49.840 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:09:49 smithi162 systemd[1]: Starting Ceph rgw.foo.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:09:49.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.962+0000 7f79a8ff9700 1 -- 172.21.15.181:0/1950307810 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v60) ==== 160+0+0 (unknown 701681216 0 0) 0x7f7994056830 con 0x7f79ac1064a0 2024-04-27T01:09:49.955 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.965+0000 7f79b1360700 1 -- 172.21.15.181:0/1950307810 >> v1:172.21.15.162:6800/2166195007 conn(0x7f799806ee20 legacy=0x7f79980712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:49.955 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.965+0000 7f79b1360700 1 -- 172.21.15.181:0/1950307810 >> v1:172.21.15.181:6789/0 conn(0x7f79ac1064a0 legacy=0x7f79ac19f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:49.955 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.965+0000 7f79b1360700 1 -- 172.21.15.181:0/1950307810 shutdown_connections 2024-04-27T01:09:49.955 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.965+0000 7f79b1360700 1 -- 172.21.15.181:0/1950307810 >> 172.21.15.181:0/1950307810 conn(0x7f79ac0fd630 msgr2=0x7f79ac1088f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:49.955 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.965+0000 7f79b1360700 1 -- 172.21.15.181:0/1950307810 shutdown_connections 2024-04-27T01:09:49.955 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:49.965+0000 7f79b1360700 1 -- 172.21.15.181:0/1950307810 wait complete. 2024-04-27T01:09:49.956 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'datapool' created 2024-04-27T01:09:50.133 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:49 smithi162 ceph-mon[26714]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-27T01:09:50.133 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/1950307810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-27T01:09:50.133 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:49 smithi162 ceph-mon[26714]: from='client.24364 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-27T01:09:50.133 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:49 smithi162 ceph-mon[37680]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-27T01:09:50.133 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/1950307810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-27T01:09:50.133 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:49 smithi162 ceph-mon[37680]: from='client.24364 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-27T01:09:50.134 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:09:49 smithi162 podman[63929]: 2024-04-27 01:09:49.7651009 +0000 UTC m=+0.018074906 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:09:50.134 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:09:49 smithi162 podman[63929]: 2024-04-27 01:09:49.864709345 +0000 UTC m=+0.117683318 container create b6dba62f254e72e3e9b351c6e9960e83597ca043608eb07b1ba02c1a99464456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-27T01:09:50.134 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:09:50 smithi162 podman[63929]: 2024-04-27 01:09:50.084512433 +0000 UTC m=+0.337486408 container init b6dba62f254e72e3e9b351c6e9960e83597ca043608eb07b1ba02c1a99464456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:09:50.134 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:09:50 smithi162 podman[63929]: 2024-04-27 01:09:50.087041136 +0000 UTC m=+0.340015118 container start b6dba62f254e72e3e9b351c6e9960e83597ca043608eb07b1ba02c1a99464456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-27T01:09:50.134 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:09:50 smithi162 bash[63929]: b6dba62f254e72e3e9b351c6e9960e83597ca043608eb07b1ba02c1a99464456 2024-04-27T01:09:50.251 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:49 smithi181 ceph-mon[30237]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-27T01:09:50.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/1950307810' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-27T01:09:50.252 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:49 smithi181 ceph-mon[30237]: from='client.24364 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-27T01:09:50.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- rbd pool init datapool 2024-04-27T01:09:50.589 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:09:50 smithi162 systemd[1]: Started Ceph rgw.foo.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:09:50.793 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:09:51.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:50 smithi181 ceph-mon[30237]: from='client.24364 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-27T01:09:51.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:50 smithi181 ceph-mon[30237]: osdmap e60: 8 total, 8 up, 8 in 2024-04-27T01:09:51.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:50 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:50 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:50 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:50 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:50 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:50 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:51.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[37680]: from='client.24364 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-27T01:09:51.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[37680]: osdmap e60: 8 total, 8 up, 8 in 2024-04-27T01:09:51.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.279 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.280 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:51.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[26714]: from='client.24364 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-27T01:09:51.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[26714]: osdmap e60: 8 total, 8 up, 8 in 2024-04-27T01:09:51.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:51.280 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:50 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:09:52.087 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:51 smithi181 ceph-mon[30237]: Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-04-27T01:09:52.087 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:51 smithi181 ceph-mon[30237]: osdmap e61: 8 total, 8 up, 8 in 2024-04-27T01:09:52.087 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/62402443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-27T01:09:52.087 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:51 smithi181 ceph-mon[30237]: pgmap v186: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T01:09:52.087 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:51 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:52.088 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/62402443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-27T01:09:52.088 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:51 smithi181 ceph-mon[30237]: osdmap e62: 8 total, 8 up, 8 in 2024-04-27T01:09:52.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[37680]: Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-04-27T01:09:52.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[37680]: osdmap e61: 8 total, 8 up, 8 in 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/62402443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[37680]: pgmap v186: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/62402443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[37680]: osdmap e62: 8 total, 8 up, 8 in 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[26714]: Saving service rgw.foo.a spec with placement smithi162=foo.a;count:1 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[26714]: osdmap e61: 8 total, 8 up, 8 in 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/62402443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[26714]: pgmap v186: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/62402443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-27T01:09:52.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:51 smithi162 ceph-mon[26714]: osdmap e62: 8 total, 8 up, 8 in 2024-04-27T01:09:53.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:09:53.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/4154690384' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-27T01:09:53.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: from='client.24388 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-27T01:09:53.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:53.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:53.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:53.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:53.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-27T01:09:53.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: osdmap e63: 8 total, 8 up, 8 in 2024-04-27T01:09:53.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-27T01:09:53.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-27T01:09:53.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:52 smithi181 ceph-mon[30237]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/4154690384' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: from='client.24388 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: osdmap e63: 8 total, 8 up, 8 in 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-27T01:09:53.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[26714]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/4154690384' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: from='client.24388 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: osdmap e63: 8 total, 8 up, 8 in 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-27T01:09:53.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:52 smithi162 ceph-mon[37680]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-27T01:09:54.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:54 smithi181 ceph-mon[30237]: Checking dashboard <-> RGW credentials 2024-04-27T01:09:54.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:54 smithi181 ceph-mon[30237]: pgmap v189: 68 pgs: 7 creating+peering, 1 active+clean, 60 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:09:54.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-27T01:09:54.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:54 smithi181 ceph-mon[30237]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-27T01:09:54.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:54 smithi181 ceph-mon[30237]: osdmap e64: 8 total, 8 up, 8 in 2024-04-27T01:09:54.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:54 smithi162 ceph-mon[26714]: Checking dashboard <-> RGW credentials 2024-04-27T01:09:54.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:54 smithi162 ceph-mon[26714]: pgmap v189: 68 pgs: 7 creating+peering, 1 active+clean, 60 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:09:54.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-27T01:09:54.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:54 smithi162 ceph-mon[26714]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-27T01:09:54.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:54 smithi162 ceph-mon[26714]: osdmap e64: 8 total, 8 up, 8 in 2024-04-27T01:09:54.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:54 smithi162 ceph-mon[37680]: Checking dashboard <-> RGW credentials 2024-04-27T01:09:54.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:54 smithi162 ceph-mon[37680]: pgmap v189: 68 pgs: 7 creating+peering, 1 active+clean, 60 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:09:54.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-27T01:09:54.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:54 smithi162 ceph-mon[37680]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-27T01:09:54.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:54 smithi162 ceph-mon[37680]: osdmap e64: 8 total, 8 up, 8 in 2024-04-27T01:09:55.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.181 --placement '1;smithi181=iscsi.a' 2024-04-27T01:09:55.790 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:09:56.235 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:55 smithi181 ceph-mon[30237]: osdmap e65: 8 total, 8 up, 8 in 2024-04-27T01:09:56.235 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-27T01:09:56.235 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-27T01:09:56.235 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:55 smithi181 ceph-mon[30237]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-27T01:09:56.235 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:55 smithi181 ceph-mon[30237]: pgmap v192: 100 pgs: 10 creating+peering, 50 active+clean, 40 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-27T01:09:56.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:55 smithi162 ceph-mon[37680]: osdmap e65: 8 total, 8 up, 8 in 2024-04-27T01:09:56.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-27T01:09:56.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-27T01:09:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:55 smithi162 ceph-mon[37680]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-27T01:09:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:55 smithi162 ceph-mon[37680]: pgmap v192: 100 pgs: 10 creating+peering, 50 active+clean, 40 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-27T01:09:56.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:55 smithi162 ceph-mon[26714]: osdmap e65: 8 total, 8 up, 8 in 2024-04-27T01:09:56.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-27T01:09:56.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-27T01:09:56.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:55 smithi162 ceph-mon[26714]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-27T01:09:56.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:55 smithi162 ceph-mon[26714]: pgmap v192: 100 pgs: 10 creating+peering, 50 active+clean, 40 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-27T01:09:57.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-27T01:09:57.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:56 smithi181 ceph-mon[30237]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-27T01:09:57.195 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:56 smithi181 ceph-mon[30237]: osdmap e66: 8 total, 8 up, 8 in 2024-04-27T01:09:57.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-27T01:09:57.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:56 smithi162 ceph-mon[37680]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-27T01:09:57.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:56 smithi162 ceph-mon[37680]: osdmap e66: 8 total, 8 up, 8 in 2024-04-27T01:09:57.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-27T01:09:57.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:56 smithi162 ceph-mon[26714]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-27T01:09:57.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:56 smithi162 ceph-mon[26714]: osdmap e66: 8 total, 8 up, 8 in 2024-04-27T01:09:57.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.416+0000 7fda57ea6700 1 -- 172.21.15.181:0/2733224682 >> v1:172.21.15.181:6789/0 conn(0x7fda50100310 legacy=0x7fda50100730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:57.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.416+0000 7fda57ea6700 1 -- 172.21.15.181:0/2733224682 shutdown_connections 2024-04-27T01:09:57.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.416+0000 7fda57ea6700 1 -- 172.21.15.181:0/2733224682 >> 172.21.15.181:0/2733224682 conn(0x7fda500fbcd0 msgr2=0x7fda500fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:57.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.416+0000 7fda57ea6700 1 -- 172.21.15.181:0/2733224682 shutdown_connections 2024-04-27T01:09:57.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.416+0000 7fda57ea6700 1 -- 172.21.15.181:0/2733224682 wait complete. 2024-04-27T01:09:57.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.417+0000 7fda57ea6700 1 Processor -- start 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.417+0000 7fda57ea6700 1 -- start start 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.417+0000 7fda57ea6700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fda50198140 con 0x7fda50106520 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.417+0000 7fda57ea6700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fda501a3930 con 0x7fda5010a180 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.417+0000 7fda57ea6700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fda501a4b10 con 0x7fda50100310 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.417+0000 7fda56443700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fda5010a180 0x7fda501a1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:47994/0 (socket says 172.21.15.181:47994) 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.417+0000 7fda56443700 1 -- 172.21.15.181:0/444982812 learned_addr learned my addr 172.21.15.181:0/444982812 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.417+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983991594 0 0) 0x7fda501a3930 con 0x7fda5010a180 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda3c0035b0 con 0x7fda5010a180 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1071540314 0 0) 0x7fda50198140 con 0x7fda50106520 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda501a3930 con 0x7fda50106520 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068755986 0 0) 0x7fda501a4b10 con 0x7fda50100310 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda50198140 con 0x7fda50100310 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3874282853 0 0) 0x7fda3c0035b0 con 0x7fda5010a180 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda501a4b10 con 0x7fda5010a180 2024-04-27T01:09:57.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fda48001c70 con 0x7fda5010a180 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3327943714 0 0) 0x7fda501a3930 con 0x7fda50106520 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda3c0035b0 con 0x7fda50106520 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fda40001870 con 0x7fda50106520 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4084211210 0 0) 0x7fda50198140 con 0x7fda50100310 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda501a3930 con 0x7fda50100310 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fda4c0018f0 con 0x7fda50100310 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3120329591 0 0) 0x7fda501a4b10 con 0x7fda5010a180 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 >> v1:172.21.15.162:6790/0 conn(0x7fda50100310 legacy=0x7fda501975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 >> v1:172.21.15.162:6789/0 conn(0x7fda50106520 legacy=0x7fda5019dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda501a5d30 con 0x7fda5010a180 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda57ea6700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda501a4d40 con 0x7fda5010a180 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda57ea6700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fda501a5280 con 0x7fda5010a180 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fda48002040 con 0x7fda5010a180 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.418+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fda48006070 con 0x7fda5010a180 2024-04-27T01:09:57.412 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.419+0000 7fda57ea6700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda34005150 con 0x7fda5010a180 2024-04-27T01:09:57.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.420+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7fda480043f0 con 0x7fda5010a180 2024-04-27T01:09:57.413 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.420+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(67..67 src has 1..67) ==== 5915+0+0 (unknown 285392756 0 0) 0x7fda480896b0 con 0x7fda5010a180 2024-04-27T01:09:57.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.425+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fda48056f70 con 0x7fda5010a180 2024-04-27T01:09:57.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.635+0000 7fda57ea6700 1 -- 172.21.15.181:0/444982812 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.181", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}) -- 0x7fda34000c90 con 0x7fda3c06ee50 2024-04-27T01:09:57.634 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.643+0000 7fda46ffd700 1 -- 172.21.15.181:0/444982812 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+35 (unknown 0 0 803663096) 0x7fda34000c90 con 0x7fda3c06ee50 2024-04-27T01:09:57.635 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled iscsi.datapool update... 2024-04-27T01:09:57.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.646+0000 7fda57ea6700 1 -- 172.21.15.181:0/444982812 >> v1:172.21.15.162:6800/2166195007 conn(0x7fda3c06ee50 legacy=0x7fda3c071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:57.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.646+0000 7fda57ea6700 1 -- 172.21.15.181:0/444982812 >> v1:172.21.15.181:6789/0 conn(0x7fda5010a180 legacy=0x7fda501a1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:09:57.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.646+0000 7fda57ea6700 1 -- 172.21.15.181:0/444982812 shutdown_connections 2024-04-27T01:09:57.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.646+0000 7fda57ea6700 1 -- 172.21.15.181:0/444982812 >> 172.21.15.181:0/444982812 conn(0x7fda500fbcd0 msgr2=0x7fda500fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:09:57.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.646+0000 7fda57ea6700 1 -- 172.21.15.181:0/444982812 shutdown_connections 2024-04-27T01:09:57.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:09:57.646+0000 7fda57ea6700 1 -- 172.21.15.181:0/444982812 wait complete. 2024-04-27T01:09:58.204 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:57 smithi181 ceph-mon[30237]: osdmap e67: 8 total, 8 up, 8 in 2024-04-27T01:09:58.204 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-27T01:09:58.205 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-27T01:09:58.205 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:57 smithi181 ceph-mon[30237]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-27T01:09:58.205 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:57 smithi181 ceph-mon[30237]: pgmap v195: 132 pgs: 8 creating+peering, 69 active+clean, 55 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-27T01:09:58.205 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:57 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:58.206 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-04-27T01:09:58.206 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:09:58.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-27T01:09:58.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[37680]: osdmap e67: 8 total, 8 up, 8 in 2024-04-27T01:09:58.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-27T01:09:58.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-27T01:09:58.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[37680]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-27T01:09:58.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[37680]: pgmap v195: 132 pgs: 8 creating+peering, 69 active+clean, 55 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-27T01:09:58.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:58.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[26714]: osdmap e67: 8 total, 8 up, 8 in 2024-04-27T01:09:58.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-27T01:09:58.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-27T01:09:58.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[26714]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-27T01:09:58.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[26714]: pgmap v195: 132 pgs: 8 creating+peering, 69 active+clean, 55 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-27T01:09:58.234 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:57 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:09:58.238 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:09:58.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-27T01:09:58.264 DEBUG:teuthology.orchestra.run.smithi181:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@iscsi.iscsi.a.service 2024-04-27T01:09:58.306 INFO:tasks.cephadm:Adding prometheus.a on smithi181 2024-04-27T01:09:58.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch apply prometheus '1;smithi181=a' 2024-04-27T01:09:58.579 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:09:59.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:58 smithi181 ceph-mon[30237]: from='client.24421 v1:172.21.15.181:0/444982812' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.181", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:59.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:58 smithi181 ceph-mon[30237]: Saving service iscsi.datapool spec with placement smithi181=iscsi.a;count:1 2024-04-27T01:09:59.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-27T01:09:59.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:58 smithi181 ceph-mon[30237]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-27T01:09:59.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:58 smithi181 ceph-mon[30237]: osdmap e68: 8 total, 8 up, 8 in 2024-04-27T01:09:59.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-27T01:09:59.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:58 smithi181 ceph-mon[30237]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-27T01:09:59.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-27T01:09:59.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:58 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:09:59.339 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a[63945]: 2024-04-27T01:09:59.040+0000 7f0340b91ac0 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-27T01:09:59.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[26714]: from='client.24421 v1:172.21.15.181:0/444982812' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.181", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[26714]: Saving service iscsi.datapool spec with placement smithi181=iscsi.a;count:1 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[26714]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[26714]: osdmap e68: 8 total, 8 up, 8 in 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[26714]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[37680]: from='client.24421 v1:172.21.15.181:0/444982812' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.181", "placement": "1;smithi181=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[37680]: Saving service iscsi.datapool spec with placement smithi181=iscsi.a;count:1 2024-04-27T01:09:59.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-27T01:09:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[37680]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-27T01:09:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[37680]: osdmap e68: 8 total, 8 up, 8 in 2024-04-27T01:09:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1904240865' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-27T01:09:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[37680]: from='client.34367 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-27T01:09:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-27T01:09:59.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:58 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:10:00.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.193+0000 7f2721ae3700 1 -- 172.21.15.181:0/1317645056 >> v1:172.21.15.181:6789/0 conn(0x7f271c10a1c0 legacy=0x7f271c10c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:00.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.193+0000 7f2721ae3700 1 -- 172.21.15.181:0/1317645056 shutdown_connections 2024-04-27T01:10:00.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.193+0000 7f2721ae3700 1 -- 172.21.15.181:0/1317645056 >> 172.21.15.181:0/1317645056 conn(0x7f271c0fc520 msgr2=0x7f271c0fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:00.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.193+0000 7f2721ae3700 1 -- 172.21.15.181:0/1317645056 shutdown_connections 2024-04-27T01:10:00.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.193+0000 7f2721ae3700 1 -- 172.21.15.181:0/1317645056 wait complete. 2024-04-27T01:10:00.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.194+0000 7f2721ae3700 1 Processor -- start 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.194+0000 7f2721ae3700 1 -- start start 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.194+0000 7f2721ae3700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f271c198140 con 0x7f271c102b10 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.194+0000 7f2721ae3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f271c1a3930 con 0x7f271c10a1c0 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.194+0000 7f2721ae3700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f271c1a4b10 con 0x7f271c106620 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f271bfff700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f271c10a1c0 0x7f271c1a1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:48018/0 (socket says 172.21.15.181:48018) 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f271bfff700 1 -- 172.21.15.181:0/1216253050 learned_addr learned my addr 172.21.15.181:0/1216253050 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1265361678 0 0) 0x7f271c1a3930 con 0x7f271c10a1c0 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27040035b0 con 0x7f271c10a1c0 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3209318191 0 0) 0x7f271c1a4b10 con 0x7f271c106620 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f271c1a3930 con 0x7f271c106620 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 321606516 0 0) 0x7f271c198140 con 0x7f271c102b10 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f271c1a4b10 con 0x7f271c102b10 2024-04-27T01:10:00.188 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1539410824 0 0) 0x7f27040035b0 con 0x7f271c10a1c0 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f271c198140 con 0x7f271c10a1c0 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2714004af0 con 0x7f271c10a1c0 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2754989960 0 0) 0x7f271c1a3930 con 0x7f271c106620 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f27040035b0 con 0x7f271c106620 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2710001830 con 0x7f271c106620 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4105010595 0 0) 0x7f271c1a4b10 con 0x7f271c102b10 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f271c1a3930 con 0x7f271c102b10 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f270c001830 con 0x7f271c102b10 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2732387422 0 0) 0x7f271c198140 con 0x7f271c10a1c0 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 >> v1:172.21.15.162:6790/0 conn(0x7f271c106620 legacy=0x7f271c19dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 >> v1:172.21.15.162:6789/0 conn(0x7f271c102b10 legacy=0x7f271c1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271c1a5d30 con 0x7f271c10a1c0 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2721ae3700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f271c1a4d40 con 0x7f271c10a1c0 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.195+0000 7f2721ae3700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f271c1a5280 con 0x7f271c10a1c0 2024-04-27T01:10:00.189 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.196+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f2714004860 con 0x7f271c10a1c0 2024-04-27T01:10:00.190 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.196+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2714005120 con 0x7f271c10a1c0 2024-04-27T01:10:00.190 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.196+0000 7f2721ae3700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2708005150 con 0x7f271c10a1c0 2024-04-27T01:10:00.190 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.197+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f27140052c0 con 0x7f271c10a1c0 2024-04-27T01:10:00.190 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.197+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(69..69 src has 1..69) ==== 5942+0+0 (unknown 2145615143 0 0) 0x7f27140898f0 con 0x7f271c10a1c0 2024-04-27T01:10:00.192 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.202+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f2714057190 con 0x7f271c10a1c0 2024-04-27T01:10:00.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:59 smithi181 ceph-mon[30237]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-27T01:10:00.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-27T01:10:00.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:59 smithi181 ceph-mon[30237]: osdmap e69: 8 total, 8 up, 8 in 2024-04-27T01:10:00.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:59 smithi181 ceph-mon[30237]: pgmap v198: 132 pgs: 5 creating+peering, 75 active+clean, 52 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:10:00.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:59 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:00.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:59 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:00.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:59 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:00.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:59 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-27T01:10:00.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:59 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-27T01:10:00.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:09:59 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:00.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[37680]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[37680]: osdmap e69: 8 total, 8 up, 8 in 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[37680]: pgmap v198: 132 pgs: 5 creating+peering, 75 active+clean, 52 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[26714]: from='client.34367 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-27T01:10:00.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/887849892' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-27T01:10:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[26714]: osdmap e69: 8 total, 8 up, 8 in 2024-04-27T01:10:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[26714]: pgmap v198: 132 pgs: 5 creating+peering, 75 active+clean, 52 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:10:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-27T01:10:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-27T01:10:00.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:09:59 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:00.408 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.417+0000 7f2721ae3700 1 -- 172.21.15.181:0/1216253050 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}) -- 0x7f2708000bc0 con 0x7f270406ef90 2024-04-27T01:10:00.415 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.425+0000 7f2718ff9700 1 -- 172.21.15.181:0/1216253050 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+31 (unknown 0 0 1342662408) 0x7f2708000bc0 con 0x7f270406ef90 2024-04-27T01:10:00.416 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled prometheus update... 2024-04-27T01:10:00.418 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.427+0000 7f2721ae3700 1 -- 172.21.15.181:0/1216253050 >> v1:172.21.15.162:6800/2166195007 conn(0x7f270406ef90 legacy=0x7f2704071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:00.418 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.427+0000 7f2721ae3700 1 -- 172.21.15.181:0/1216253050 >> v1:172.21.15.181:6789/0 conn(0x7f271c10a1c0 legacy=0x7f271c1a1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:00.418 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.427+0000 7f2721ae3700 1 -- 172.21.15.181:0/1216253050 shutdown_connections 2024-04-27T01:10:00.418 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.427+0000 7f2721ae3700 1 -- 172.21.15.181:0/1216253050 >> 172.21.15.181:0/1216253050 conn(0x7f271c0fc520 msgr2=0x7f271c0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:00.418 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.427+0000 7f2721ae3700 1 -- 172.21.15.181:0/1216253050 shutdown_connections 2024-04-27T01:10:00.418 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:00.427+0000 7f2721ae3700 1 -- 172.21.15.181:0/1216253050 wait complete. 2024-04-27T01:10:01.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:00 smithi181 ceph-mon[30237]: Deploying daemon iscsi.iscsi.a on smithi181 2024-04-27T01:10:01.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:00 smithi181 ceph-mon[30237]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-27T01:10:01.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:00 smithi181 ceph-mon[30237]: Cluster is now healthy 2024-04-27T01:10:01.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:00 smithi181 ceph-mon[30237]: overall HEALTH_OK 2024-04-27T01:10:01.111 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:00 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:01.139 DEBUG:teuthology.orchestra.run.smithi181:prometheus.a> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@prometheus.a.service 2024-04-27T01:10:01.141 INFO:tasks.cephadm:Adding node-exporter.a on smithi162 2024-04-27T01:10:01.141 INFO:tasks.cephadm:Adding node-exporter.b on smithi181 2024-04-27T01:10:01.142 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch apply node-exporter '2;smithi162=a;smithi181=b' 2024-04-27T01:10:01.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:00 smithi162 ceph-mon[26714]: Deploying daemon iscsi.iscsi.a on smithi181 2024-04-27T01:10:01.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:00 smithi162 ceph-mon[26714]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-27T01:10:01.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:00 smithi162 ceph-mon[26714]: Cluster is now healthy 2024-04-27T01:10:01.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:00 smithi162 ceph-mon[26714]: overall HEALTH_OK 2024-04-27T01:10:01.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:00 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:01.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:00 smithi162 ceph-mon[37680]: Deploying daemon iscsi.iscsi.a on smithi181 2024-04-27T01:10:01.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:00 smithi162 ceph-mon[37680]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-27T01:10:01.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:00 smithi162 ceph-mon[37680]: Cluster is now healthy 2024-04-27T01:10:01.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:00 smithi162 ceph-mon[37680]: overall HEALTH_OK 2024-04-27T01:10:01.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:00 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:02.104 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:10:02.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:02 smithi181 ceph-mon[30237]: from='client.24442 v1:172.21.15.181:0/1216253050' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:02.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:02 smithi181 ceph-mon[30237]: Saving service prometheus spec with placement smithi181=a;count:1 2024-04-27T01:10:02.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:02 smithi181 ceph-mon[30237]: pgmap v199: 132 pgs: 125 active+clean, 7 unknown; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 5.8 KiB/s wr, 206 op/s 2024-04-27T01:10:02.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:02 smithi162 ceph-mon[26714]: from='client.24442 v1:172.21.15.181:0/1216253050' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:02.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:02 smithi162 ceph-mon[26714]: Saving service prometheus spec with placement smithi181=a;count:1 2024-04-27T01:10:02.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:02 smithi162 ceph-mon[26714]: pgmap v199: 132 pgs: 125 active+clean, 7 unknown; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 5.8 KiB/s wr, 206 op/s 2024-04-27T01:10:02.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:02 smithi162 ceph-mon[37680]: from='client.24442 v1:172.21.15.181:0/1216253050' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:02.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:02 smithi162 ceph-mon[37680]: Saving service prometheus spec with placement smithi181=a;count:1 2024-04-27T01:10:02.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:02 smithi162 ceph-mon[37680]: pgmap v199: 132 pgs: 125 active+clean, 7 unknown; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 5.8 KiB/s wr, 206 op/s 2024-04-27T01:10:02.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:02 smithi181 systemd[1]: Starting Ceph iscsi.iscsi.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:10:03.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.896+0000 7f45430a3700 1 -- 172.21.15.181:0/3146513322 >> v1:172.21.15.181:6789/0 conn(0x7f453c1064e0 legacy=0x7f453c108930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:03.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.896+0000 7f45430a3700 1 -- 172.21.15.181:0/3146513322 shutdown_connections 2024-04-27T01:10:03.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.896+0000 7f45430a3700 1 -- 172.21.15.181:0/3146513322 >> 172.21.15.181:0/3146513322 conn(0x7f453c0fd650 msgr2=0x7f453c0ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:03.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.897+0000 7f45430a3700 1 -- 172.21.15.181:0/3146513322 shutdown_connections 2024-04-27T01:10:03.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.897+0000 7f45430a3700 1 -- 172.21.15.181:0/3146513322 wait complete. 2024-04-27T01:10:03.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.897+0000 7f45430a3700 1 Processor -- start 2024-04-27T01:10:03.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.897+0000 7f45430a3700 1 -- start start 2024-04-27T01:10:03.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.897+0000 7f45430a3700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f453c1981b0 con 0x7f453c1020b0 2024-04-27T01:10:03.890 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.897+0000 7f45430a3700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f453c1a39a0 con 0x7f453c1064e0 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f45430a3700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f453c1a4b80 con 0x7f453c10a220 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f453bfff700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f453c1064e0 0x7f453c19db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34098/0 (socket says 172.21.15.181:34098) 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f453bfff700 1 -- 172.21.15.181:0/1816607608 learned_addr learned my addr 172.21.15.181:0/1816607608 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 866052330 0 0) 0x7f453c1a39a0 con 0x7f453c1064e0 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f452c0035b0 con 0x7f453c1064e0 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 975248499 0 0) 0x7f453c1981b0 con 0x7f453c1020b0 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f453c1a39a0 con 0x7f453c1020b0 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3592347333 0 0) 0x7f452c0035b0 con 0x7f453c1064e0 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f453c1981b0 con 0x7f453c1064e0 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 424107805 0 0) 0x7f453c1a4b80 con 0x7f453c10a220 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f452c0035b0 con 0x7f453c10a220 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f4528001430 con 0x7f453c1064e0 2024-04-27T01:10:03.891 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1232147018 0 0) 0x7f453c1a39a0 con 0x7f453c1020b0 2024-04-27T01:10:03.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f453c1a4b80 con 0x7f453c1020b0 2024-04-27T01:10:03.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f45300017e0 con 0x7f453c1020b0 2024-04-27T01:10:03.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2488971900 0 0) 0x7f453c1981b0 con 0x7f453c1064e0 2024-04-27T01:10:03.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 >> v1:172.21.15.162:6790/0 conn(0x7f453c10a220 legacy=0x7f453c1a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:03.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 >> v1:172.21.15.162:6789/0 conn(0x7f453c1020b0 legacy=0x7f453c197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:03.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f453c1a5da0 con 0x7f453c1064e0 2024-04-27T01:10:03.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.898+0000 7f45430a3700 1 -- 172.21.15.181:0/1816607608 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f453c1a29f0 con 0x7f453c1064e0 2024-04-27T01:10:03.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.899+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f4528001650 con 0x7f453c1064e0 2024-04-27T01:10:03.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.899+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f45280050d0 con 0x7f453c1064e0 2024-04-27T01:10:03.892 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.899+0000 7f45430a3700 1 -- 172.21.15.181:0/1816607608 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f453c1a2f30 con 0x7f453c1064e0 2024-04-27T01:10:03.893 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.900+0000 7f45430a3700 1 -- 172.21.15.181:0/1816607608 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f453c068b50 con 0x7f453c1064e0 2024-04-27T01:10:03.893 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.900+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 15) ==== 92082+0+0 (unknown 2438049355 0 0) 0x7f45280050b0 con 0x7f453c1064e0 2024-04-27T01:10:03.893 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.901+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(70..70 src has 1..70) ==== 5942+0+0 (unknown 665183937 0 0) 0x7f4528089590 con 0x7f453c1064e0 2024-04-27T01:10:03.896 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:03.905+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f4528056dd0 con 0x7f453c1064e0 2024-04-27T01:10:04.125 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:04.134+0000 7f45430a3700 1 -- 172.21.15.181:0/1816607608 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi162=a;smithi181=b", "target": ["mon-mgr", ""]}) -- 0x7f453c1a32c0 con 0x7f452c06ebe0 2024-04-27T01:10:04.132 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:04.141+0000 7f4539ffb700 1 -- 172.21.15.181:0/1816607608 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+34 (unknown 0 0 240551134) 0x7f453c1a32c0 con 0x7f452c06ebe0 2024-04-27T01:10:04.132 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled node-exporter update... 2024-04-27T01:10:04.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:04.143+0000 7f45430a3700 1 -- 172.21.15.181:0/1816607608 >> v1:172.21.15.162:6800/2166195007 conn(0x7f452c06ebe0 legacy=0x7f452c0710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:04.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:04.143+0000 7f45430a3700 1 -- 172.21.15.181:0/1816607608 >> v1:172.21.15.181:6789/0 conn(0x7f453c1064e0 legacy=0x7f453c19db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:04.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:04.143+0000 7f45430a3700 1 -- 172.21.15.181:0/1816607608 shutdown_connections 2024-04-27T01:10:04.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:04.143+0000 7f45430a3700 1 -- 172.21.15.181:0/1816607608 >> 172.21.15.181:0/1816607608 conn(0x7f453c0fd650 msgr2=0x7f453c108930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:04.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:04.143+0000 7f45430a3700 1 -- 172.21.15.181:0/1816607608 shutdown_connections 2024-04-27T01:10:04.134 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:04.143+0000 7f45430a3700 1 -- 172.21.15.181:0/1816607608 wait complete. 2024-04-27T01:10:04.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:04 smithi181 ceph-mon[30237]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 5.8 KiB/s wr, 235 op/s 2024-04-27T01:10:04.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:04 smithi181 ceph-mon[30237]: osdmap e70: 8 total, 8 up, 8 in 2024-04-27T01:10:04.208 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:04 smithi181 podman[57437]: 2024-04-27 01:10:03.966624761 +0000 UTC m=+0.450852154 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:10:04.208 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:04 smithi181 podman[57437]: 2024-04-27 01:10:04.096870357 +0000 UTC m=+0.581097775 container create bafd375641d6267be09baa78894f863e69acbdef858f24f2040c62f56d106c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:10:04.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:04 smithi162 ceph-mon[26714]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 5.8 KiB/s wr, 235 op/s 2024-04-27T01:10:04.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:04 smithi162 ceph-mon[26714]: osdmap e70: 8 total, 8 up, 8 in 2024-04-27T01:10:04.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:04 smithi162 ceph-mon[37680]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 5.8 KiB/s wr, 235 op/s 2024-04-27T01:10:04.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:04 smithi162 ceph-mon[37680]: osdmap e70: 8 total, 8 up, 8 in 2024-04-27T01:10:04.837 DEBUG:teuthology.orchestra.run.smithi162:node-exporter.a> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@node-exporter.a.service 2024-04-27T01:10:04.840 DEBUG:teuthology.orchestra.run.smithi181:node-exporter.b> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@node-exporter.b.service 2024-04-27T01:10:04.842 INFO:tasks.cephadm:Adding alertmanager.a on smithi162 2024-04-27T01:10:04.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch apply alertmanager '1;smithi162=a' 2024-04-27T01:10:05.031 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:04 smithi181 podman[57437]: 2024-04-27 01:10:04.966984426 +0000 UTC m=+1.451211800 container init bafd375641d6267be09baa78894f863e69acbdef858f24f2040c62f56d106c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:10:05.032 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:04 smithi181 podman[57437]: 2024-04-27 01:10:04.971712693 +0000 UTC m=+1.455940059 container start bafd375641d6267be09baa78894f863e69acbdef858f24f2040c62f56d106c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-04-27T01:10:05.147 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:10:05.374 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-mon[30237]: from='client.24448 v1:172.21.15.181:0/1816607608' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi162=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:05.374 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-mon[30237]: Saving service node-exporter spec with placement smithi162=a;smithi181=b;count:2 2024-04-27T01:10:05.374 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:05.374 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:05.374 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:05.374 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:05 smithi181 bash[57437]: bafd375641d6267be09baa78894f863e69acbdef858f24f2040c62f56d106c94 2024-04-27T01:10:05.374 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:05 smithi181 systemd[1]: Started Ceph iscsi.iscsi.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:10:05.375 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug Started the configuration object watcher 2024-04-27T01:10:05.375 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug Checking for config object changes every 1s 2024-04-27T01:10:05.375 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug Processing osd blocklist entries for this node 2024-04-27T01:10:05.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:05 smithi162 ceph-mon[26714]: from='client.24448 v1:172.21.15.181:0/1816607608' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi162=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:05.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:05 smithi162 ceph-mon[26714]: Saving service node-exporter spec with placement smithi162=a;smithi181=b;count:2 2024-04-27T01:10:05.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:05 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:05.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:05 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:05.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:05 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:05.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:05 smithi162 ceph-mon[37680]: from='client.24448 v1:172.21.15.181:0/1816607608' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi162=a;smithi181=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:05.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:05 smithi162 ceph-mon[37680]: Saving service node-exporter spec with placement smithi162=a;smithi181=b;count:2 2024-04-27T01:10:05.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:05.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:05.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:05 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:05.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug Reading the configuration object to update local LIO configuration 2024-04-27T01:10:05.939 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug Configuration does not have an entry for this host(smithi181) - nothing to define to LIO 2024-04-27T01:10:05.939 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:05 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-04-27T01:10:06.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:06 smithi181 ceph-mon[30237]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 5.3 KiB/s wr, 228 op/s 2024-04-27T01:10:06.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:06 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:06.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:06 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:06.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:06 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:06.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/2332811109' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-27T01:10:06.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:06 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:06.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[37680]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 5.3 KiB/s wr, 228 op/s 2024-04-27T01:10:06.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:06.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:06.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:06.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/2332811109' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-27T01:10:06.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:06.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[26714]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s rd, 5.3 KiB/s wr, 228 op/s 2024-04-27T01:10:06.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:06.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:06.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:06.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/2332811109' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-27T01:10:06.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:06 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:07.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:07 smithi181 ceph-mon[30237]: Checking pool "datapool" exists for service iscsi.datapool 2024-04-27T01:10:07.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:07 smithi162 ceph-mon[26714]: Checking pool "datapool" exists for service iscsi.datapool 2024-04-27T01:10:07.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:07 smithi162 ceph-mon[37680]: Checking pool "datapool" exists for service iscsi.datapool 2024-04-27T01:10:07.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.743+0000 7f1ddc37c700 1 -- 172.21.15.181:0/3176643256 >> v1:172.21.15.181:6789/0 conn(0x7f1dd410a1a0 legacy=0x7f1dd410c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:07.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.743+0000 7f1ddc37c700 1 -- 172.21.15.181:0/3176643256 shutdown_connections 2024-04-27T01:10:07.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.743+0000 7f1ddc37c700 1 -- 172.21.15.181:0/3176643256 >> 172.21.15.181:0/3176643256 conn(0x7f1dd40fc500 msgr2=0x7f1dd40fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:07.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.743+0000 7f1ddc37c700 1 -- 172.21.15.181:0/3176643256 shutdown_connections 2024-04-27T01:10:07.734 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.743+0000 7f1ddc37c700 1 -- 172.21.15.181:0/3176643256 wait complete. 2024-04-27T01:10:07.736 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1ddc37c700 1 Processor -- start 2024-04-27T01:10:07.736 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1ddc37c700 1 -- start start 2024-04-27T01:10:07.736 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1ddc37c700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1dd4198320 con 0x7f1dd4102af0 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1ddc37c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1dd41a3b10 con 0x7f1dd410a1a0 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1ddc37c700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1dd41a4cf0 con 0x7f1dd4106600 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dda919700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f1dd410a1a0 0x7f1dd41a13e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:34182/0 (socket says 172.21.15.181:34182) 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dda919700 1 -- 172.21.15.181:0/1367734221 learned_addr learned my addr 172.21.15.181:0/1367734221 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2287232085 0 0) 0x7f1dd41a3b10 con 0x7f1dd410a1a0 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dc00035b0 con 0x7f1dd410a1a0 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1936367547 0 0) 0x7f1dd4198320 con 0x7f1dd4102af0 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dd41a3b10 con 0x7f1dd4102af0 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 91442981 0 0) 0x7f1dd41a4cf0 con 0x7f1dd4106600 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dd4198320 con 0x7f1dd4106600 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2983666710 0 0) 0x7f1dc00035b0 con 0x7f1dd410a1a0 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1dd41a4cf0 con 0x7f1dd410a1a0 2024-04-27T01:10:07.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.744+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1dcc0017b0 con 0x7f1dd410a1a0 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1779534040 0 0) 0x7f1dd4198320 con 0x7f1dd4106600 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1dc00035b0 con 0x7f1dd4106600 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4150831930 0 0) 0x7f1dd41a3b10 con 0x7f1dd4102af0 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1dd4198320 con 0x7f1dd4102af0 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1dc4001870 con 0x7f1dd4106600 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1dd0001830 con 0x7f1dd4102af0 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 904147531 0 0) 0x7f1dd41a4cf0 con 0x7f1dd410a1a0 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 >> v1:172.21.15.162:6790/0 conn(0x7f1dd4106600 legacy=0x7f1dd419dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 >> v1:172.21.15.162:6789/0 conn(0x7f1dd4102af0 legacy=0x7f1dd41977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd41a5f10 con 0x7f1dd410a1a0 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1ddc37c700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd41a2b60 con 0x7f1dd410a1a0 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1ddc37c700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1dd41a30a0 con 0x7f1dd410a1a0 2024-04-27T01:10:07.738 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f1dcc001a10 con 0x7f1dd410a1a0 2024-04-27T01:10:07.739 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.745+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1dcc005250 con 0x7f1dd410a1a0 2024-04-27T01:10:07.739 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.746+0000 7f1ddc37c700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1db8005150 con 0x7f1dd410a1a0 2024-04-27T01:10:07.739 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.746+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 16) ==== 92133+0+0 (unknown 3105418878 0 0) 0x7f1dcc003e30 con 0x7f1dd410a1a0 2024-04-27T01:10:07.739 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.747+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(70..70 src has 1..70) ==== 5942+0+0 (unknown 665183937 0 0) 0x7f1dcc0898c0 con 0x7f1dd410a1a0 2024-04-27T01:10:07.742 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:07.752+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f1dcc056fe0 con 0x7f1dd410a1a0 2024-04-27T01:10:08.017 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:08.026+0000 7f1ddc37c700 1 -- 172.21.15.181:0/1367734221 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}) -- 0x7f1db8000bc0 con 0x7f1dc006f0d0 2024-04-27T01:10:08.024 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:08.033+0000 7f1dcb7fe700 1 -- 172.21.15.181:0/1367734221 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+33 (unknown 0 0 1850065467) 0x7f1db8000bc0 con 0x7f1dc006f0d0 2024-04-27T01:10:08.024 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled alertmanager update... 2024-04-27T01:10:08.026 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:08.035+0000 7f1ddc37c700 1 -- 172.21.15.181:0/1367734221 >> v1:172.21.15.162:6800/2166195007 conn(0x7f1dc006f0d0 legacy=0x7f1dc0071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:08.026 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:08.035+0000 7f1ddc37c700 1 -- 172.21.15.181:0/1367734221 >> v1:172.21.15.181:6789/0 conn(0x7f1dd410a1a0 legacy=0x7f1dd41a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:08.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:08.035+0000 7f1ddc37c700 1 -- 172.21.15.181:0/1367734221 shutdown_connections 2024-04-27T01:10:08.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:08.035+0000 7f1ddc37c700 1 -- 172.21.15.181:0/1367734221 >> 172.21.15.181:0/1367734221 conn(0x7f1dd40fc500 msgr2=0x7f1dd40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:08.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:08.035+0000 7f1ddc37c700 1 -- 172.21.15.181:0/1367734221 shutdown_connections 2024-04-27T01:10:08.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:08.036+0000 7f1ddc37c700 1 -- 172.21.15.181:0/1367734221 wait complete. 2024-04-27T01:10:08.347 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:08 smithi181 ceph-mon[30237]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 4.5 KiB/s wr, 199 op/s 2024-04-27T01:10:08.348 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:08 smithi181 ceph-mon[30237]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-27T01:10:08.348 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:08 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:08.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:08 smithi162 ceph-mon[26714]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 4.5 KiB/s wr, 199 op/s 2024-04-27T01:10:08.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:08 smithi162 ceph-mon[26714]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-27T01:10:08.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:08 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:08.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:08 smithi162 ceph-mon[37680]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 4.5 KiB/s wr, 199 op/s 2024-04-27T01:10:08.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:08 smithi162 ceph-mon[37680]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-27T01:10:08.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:08 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:09.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:09 smithi181 ceph-mon[30237]: from='client.24484 v1:172.21.15.181:0/1367734221' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:09.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:09 smithi181 ceph-mon[30237]: Saving service alertmanager spec with placement smithi162=a;count:1 2024-04-27T01:10:09.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:09 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:09.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:09 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:09.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:09 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:09.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:09 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:10:09.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:09 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:09.337 DEBUG:teuthology.orchestra.run.smithi162:alertmanager.a> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@alertmanager.a.service 2024-04-27T01:10:09.340 INFO:tasks.cephadm:Adding grafana.a on smithi181 2024-04-27T01:10:09.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph orch apply grafana '1;smithi181=a' 2024-04-27T01:10:09.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[37680]: from='client.24484 v1:172.21.15.181:0/1367734221' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:09.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[37680]: Saving service alertmanager spec with placement smithi162=a;count:1 2024-04-27T01:10:09.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[26714]: from='client.24484 v1:172.21.15.181:0/1367734221' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi162=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[26714]: Saving service alertmanager spec with placement smithi162=a;count:1 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:10:09.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:09 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:09.608 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:10:10.425 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:10 smithi181 ceph-mon[30237]: Deploying daemon prometheus.a on smithi181 2024-04-27T01:10:10.425 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:10 smithi181 ceph-mon[30237]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 3.6 KiB/s wr, 160 op/s 2024-04-27T01:10:10.491 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:10 smithi162 ceph-mon[37680]: Deploying daemon prometheus.a on smithi181 2024-04-27T01:10:10.491 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:10 smithi162 ceph-mon[37680]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 3.6 KiB/s wr, 160 op/s 2024-04-27T01:10:10.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:10 smithi162 ceph-mon[26714]: Deploying daemon prometheus.a on smithi181 2024-04-27T01:10:10.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:10 smithi162 ceph-mon[26714]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 3.6 KiB/s wr, 160 op/s 2024-04-27T01:10:11.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.613+0000 7f4926e11700 1 -- 172.21.15.181:0/2116036784 >> v1:172.21.15.181:6789/0 conn(0x7f4920109040 legacy=0x7f492010b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.613+0000 7f4926e11700 1 -- 172.21.15.181:0/2116036784 shutdown_connections 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.613+0000 7f4926e11700 1 -- 172.21.15.181:0/2116036784 >> 172.21.15.181:0/2116036784 conn(0x7f49200fc500 msgr2=0x7f49200fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.613+0000 7f4926e11700 1 -- 172.21.15.181:0/2116036784 shutdown_connections 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.613+0000 7f4926e11700 1 -- 172.21.15.181:0/2116036784 wait complete. 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.613+0000 7f4926e11700 1 Processor -- start 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.613+0000 7f4926e11700 1 -- start start 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.613+0000 7f4926e11700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f4920197240 con 0x7f4920109040 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f4926e11700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f49201a2a30 con 0x7f49201053b0 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f4926e11700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f49201a3c10 con 0x7f4920100f10 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491ffff700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7f49201053b0 0x7f492019cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:59412/0 (socket says 172.21.15.181:59412) 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491ffff700 1 -- 172.21.15.181:0/2995328900 learned_addr learned my addr 172.21.15.181:0/2995328900 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 971960875 0 0) 0x7f49201a2a30 con 0x7f49201053b0 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49100035b0 con 0x7f49201053b0 2024-04-27T01:10:11.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198118123 0 0) 0x7f49201a3c10 con 0x7f4920100f10 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49201a2a30 con 0x7f4920100f10 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2999119299 0 0) 0x7f4920197240 con 0x7f4920109040 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49201a3c10 con 0x7f4920109040 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2361436933 0 0) 0x7f49100035b0 con 0x7f49201053b0 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4920197240 con 0x7f49201053b0 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f490c001930 con 0x7f49201053b0 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 300605841 0 0) 0x7f49201a3c10 con 0x7f4920109040 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49100035b0 con 0x7f4920109040 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f49180017b0 con 0x7f4920109040 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3491578011 0 0) 0x7f49201a2a30 con 0x7f4920100f10 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f49201a3c10 con 0x7f4920100f10 2024-04-27T01:10:11.630 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f4914001830 con 0x7f4920100f10 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 859249128 0 0) 0x7f4920197240 con 0x7f49201053b0 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 >> v1:172.21.15.162:6790/0 conn(0x7f4920100f10 legacy=0x7f49201966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 >> v1:172.21.15.162:6789/0 conn(0x7f4920109040 legacy=0x7f49201a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.614+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49201a4e30 con 0x7f49201053b0 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.615+0000 7f4926e11700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f49201a3e40 con 0x7f49201053b0 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.615+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f490c003b30 con 0x7f49201053b0 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.615+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f490c005cc0 con 0x7f49201053b0 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.615+0000 7f4926e11700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f49201a4380 con 0x7f49201053b0 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.616+0000 7f4926e11700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f492010eaf0 con 0x7f49201053b0 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.616+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 16) ==== 92133+0+0 (unknown 3105418878 0 0) 0x7f490c01c4b0 con 0x7f49201053b0 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.616+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(70..70 src has 1..70) ==== 5942+0+0 (unknown 665183937 0 0) 0x7f490c089400 con 0x7f49201053b0 2024-04-27T01:10:11.631 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.622+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f490c0036a0 con 0x7f49201053b0 2024-04-27T01:10:11.859 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.868+0000 7f4926e11700 1 -- 172.21.15.181:0/2995328900 --> v1:172.21.15.162:6800/2166195007 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}) -- 0x7f49201a4670 con 0x7f4910071380 2024-04-27T01:10:11.865 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.874+0000 7f491dffb700 1 -- 172.21.15.181:0/2995328900 <== mgr.14152 v1:172.21.15.162:6800/2166195007 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (unknown 0 0 664801700) 0x7f49201a4670 con 0x7f4910071380 2024-04-27T01:10:11.865 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled grafana update... 2024-04-27T01:10:11.869 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.877+0000 7f4926e11700 1 -- 172.21.15.181:0/2995328900 >> v1:172.21.15.162:6800/2166195007 conn(0x7f4910071380 legacy=0x7f4910073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:11.869 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.877+0000 7f4926e11700 1 -- 172.21.15.181:0/2995328900 >> v1:172.21.15.181:6789/0 conn(0x7f49201053b0 legacy=0x7f492019cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:11.869 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.877+0000 7f4926e11700 1 -- 172.21.15.181:0/2995328900 shutdown_connections 2024-04-27T01:10:11.869 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.877+0000 7f4926e11700 1 -- 172.21.15.181:0/2995328900 >> 172.21.15.181:0/2995328900 conn(0x7f49200fc500 msgr2=0x7f492010b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:11.869 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.878+0000 7f4926e11700 1 -- 172.21.15.181:0/2995328900 shutdown_connections 2024-04-27T01:10:11.869 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:11.878+0000 7f4926e11700 1 -- 172.21.15.181:0/2995328900 wait complete. 2024-04-27T01:10:12.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:12 smithi181 ceph-mon[30237]: pgmap v205: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 921 B/s wr, 58 op/s 2024-04-27T01:10:12.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:12 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:12.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:12 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:12.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:12 smithi162 ceph-mon[37680]: pgmap v205: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 921 B/s wr, 58 op/s 2024-04-27T01:10:12.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:12 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:12.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:12 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:12.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:12 smithi162 ceph-mon[26714]: pgmap v205: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 921 B/s wr, 58 op/s 2024-04-27T01:10:12.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:12 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:12.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:12 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:12.957 DEBUG:teuthology.orchestra.run.smithi181:grafana.a> sudo journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@grafana.a.service 2024-04-27T01:10:12.959 INFO:tasks.cephadm:Setting up client nodes... 2024-04-27T01:10:12.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-27T01:10:13.201 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:13.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:13 smithi181 ceph-mon[30237]: from='client.24490 v1:172.21.15.181:0/2995328900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:13.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:13 smithi181 ceph-mon[30237]: Saving service grafana spec with placement smithi181=a;count:1 2024-04-27T01:10:13.319 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:13 smithi162 ceph-mon[26714]: from='client.24490 v1:172.21.15.181:0/2995328900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:13.319 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:13 smithi162 ceph-mon[26714]: Saving service grafana spec with placement smithi181=a;count:1 2024-04-27T01:10:13.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:13 smithi162 ceph-mon[37680]: from='client.24490 v1:172.21.15.181:0/2995328900' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi181=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T01:10:13.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:13 smithi162 ceph-mon[37680]: Saving service grafana spec with placement smithi181=a;count:1 2024-04-27T01:10:14.322 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:14 smithi162 ceph-mon[37680]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 307 B/s wr, 20 op/s 2024-04-27T01:10:14.323 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:14 smithi162 ceph-mon[26714]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 307 B/s wr, 20 op/s 2024-04-27T01:10:14.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:14 smithi181 ceph-mon[30237]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 8.5 KiB/s rd, 307 B/s wr, 20 op/s 2024-04-27T01:10:14.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.967+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3911325942 >> v1:172.21.15.162:6789/0 conn(0x7fb0c0106630 legacy=0x7fb0c0108a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:14.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.967+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3911325942 shutdown_connections 2024-04-27T01:10:14.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.967+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3911325942 >> 172.21.15.162:0/3911325942 conn(0x7fb0c00fc4d0 msgr2=0x7fb0c00fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:14.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.967+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3911325942 shutdown_connections 2024-04-27T01:10:14.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.967+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3911325942 wait complete. 2024-04-27T01:10:14.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.968+0000 7fb0c4f29700 1 Processor -- start 2024-04-27T01:10:14.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.968+0000 7fb0c4f29700 1 -- start start 2024-04-27T01:10:14.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.968+0000 7fb0c4f29700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0c01a4640 con 0x7fb0c0102b20 2024-04-27T01:10:14.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.968+0000 7fb0c4f29700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0c01a5880 con 0x7fb0c010a170 2024-04-27T01:10:14.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.968+0000 7fb0c4f29700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb0c01a6ac0 con 0x7fb0c0106630 2024-04-27T01:10:14.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.968+0000 7fb0be59c700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb0c0102b20 0x7fb0c01023a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:51728/0 (socket says 172.21.15.162:51728) 2024-04-27T01:10:14.959 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.968+0000 7fb0be59c700 1 -- 172.21.15.162:0/3419396467 learned_addr learned my addr 172.21.15.162:0/3419396467 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456241973 0 0) 0x7fb0c01a6ac0 con 0x7fb0c0106630 2024-04-27T01:10:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0a8005240 con 0x7fb0c0106630 2024-04-27T01:10:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3444775446 0 0) 0x7fb0c01a4640 con 0x7fb0c0102b20 2024-04-27T01:10:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0c01a6ac0 con 0x7fb0c0102b20 2024-04-27T01:10:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106286111 0 0) 0x7fb0c01a5880 con 0x7fb0c010a170 2024-04-27T01:10:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0c01a4640 con 0x7fb0c010a170 2024-04-27T01:10:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3816529445 0 0) 0x7fb0a8005240 con 0x7fb0c0106630 2024-04-27T01:10:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0c01a5880 con 0x7fb0c0106630 2024-04-27T01:10:14.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2692656521 0 0) 0x7fb0c01a6ac0 con 0x7fb0c0102b20 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0a8005240 con 0x7fb0c0102b20 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fb0b4004470 con 0x7fb0c0106630 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fb0b0001870 con 0x7fb0c0102b20 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3087855577 0 0) 0x7fb0c01a4640 con 0x7fb0c010a170 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0c01a6ac0 con 0x7fb0c010a170 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fb0b80013f0 con 0x7fb0c010a170 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 959786155 0 0) 0x7fb0c01a5880 con 0x7fb0c0106630 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 >> v1:172.21.15.181:6789/0 conn(0x7fb0c010a170 legacy=0x7fb0c01a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 >> v1:172.21.15.162:6789/0 conn(0x7fb0c0102b20 legacy=0x7fb0c01023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c01a7d00 con 0x7fb0c0106630 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c01a6cf0 con 0x7fb0c0106630 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb0c01a7230 con 0x7fb0c0106630 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fb0b40046a0 con 0x7fb0c0106630 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.969+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fb0b4004c80 con 0x7fb0c0106630 2024-04-27T01:10:14.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.970+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0a0005150 con 0x7fb0c0106630 2024-04-27T01:10:14.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.971+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 16) ==== 92133+0+0 (unknown 3105418878 0 0) 0x7fb0b401b470 con 0x7fb0c0106630 2024-04-27T01:10:14.965 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.971+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(70..70 src has 1..70) ==== 5942+0+0 (unknown 665183937 0 0) 0x7fb0b4088230 con 0x7fb0c0106630 2024-04-27T01:10:14.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:14.976+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fb0b4003860 con 0x7fb0c0106630 2024-04-27T01:10:15.244 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:15.253+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3419396467 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fb0a0005440 con 0x7fb0c0106630 2024-04-27T01:10:15.248 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:15.258+0000 7fb0af7fe700 1 -- 172.21.15.162:0/3419396467 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (unknown 57899324 0 3912696172) 0x7fb0b40559d0 con 0x7fb0c0106630 2024-04-27T01:10:15.249 INFO:teuthology.orchestra.run.smithi162.stdout:[client.0] 2024-04-27T01:10:15.249 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQB3UCxm6qs5DxAAsl1gE+ESUzswnDyBw+4bJQ== 2024-04-27T01:10:15.250 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:15.260+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3419396467 >> v1:172.21.15.162:6800/2166195007 conn(0x7fb0a806ee50 legacy=0x7fb0a8071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:15.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:15.260+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3419396467 >> v1:172.21.15.162:6790/0 conn(0x7fb0c0106630 legacy=0x7fb0c019f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:15.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:15.260+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3419396467 shutdown_connections 2024-04-27T01:10:15.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:15.260+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3419396467 >> 172.21.15.162:0/3419396467 conn(0x7fb0c00fc4d0 msgr2=0x7fb0c00fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:15.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:15.261+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3419396467 shutdown_connections 2024-04-27T01:10:15.251 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:15.261+0000 7fb0c4f29700 1 -- 172.21.15.162:0/3419396467 wait complete. 2024-04-27T01:10:15.761 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:15 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:10:15.988 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:10:15.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-27T01:10:15.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-27T01:10:16.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-27T01:10:16.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:16 smithi162 ceph-mon[26714]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 272 B/s wr, 18 op/s 2024-04-27T01:10:16.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3419396467' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:16.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:16 smithi162 ceph-mon[26714]: from='client.34484 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:16.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:16 smithi162 ceph-mon[26714]: from='client.34484 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T01:10:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:16 smithi162 ceph-mon[37680]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 272 B/s wr, 18 op/s 2024-04-27T01:10:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3419396467' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:16 smithi162 ceph-mon[37680]: from='client.34484 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:16 smithi162 ceph-mon[37680]: from='client.34484 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T01:10:16.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:16 smithi181 ceph-mon[30237]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 272 B/s wr, 18 op/s 2024-04-27T01:10:16.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3419396467' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:16.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:16 smithi181 ceph-mon[30237]: from='client.34484 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:16.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:16 smithi181 ceph-mon[30237]: from='client.34484 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T01:10:16.574 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.b/config 2024-04-27T01:10:17.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:17 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:10:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:17 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:10:17.395 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:17 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:10:18.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:18 smithi162 ceph-mon[37680]: pgmap v208: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T01:10:18.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:18 smithi162 ceph-mon[26714]: pgmap v208: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T01:10:18.485 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:18 smithi181 ceph-mon[30237]: pgmap v208: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T01:10:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:20 smithi181 ceph-mon[30237]: pgmap v209: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T01:10:20.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:20 smithi162 ceph-mon[37680]: pgmap v209: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T01:10:20.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:20 smithi162 ceph-mon[26714]: pgmap v209: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-04-27T01:10:21.026 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.035+0000 7fcb24301700 1 -- 172.21.15.181:0/1364991307 >> v1:172.21.15.181:6789/0 conn(0x7fcb1c102b50 legacy=0x7fcb1c104f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:21.026 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.035+0000 7fcb24301700 1 -- 172.21.15.181:0/1364991307 shutdown_connections 2024-04-27T01:10:21.026 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.035+0000 7fcb24301700 1 -- 172.21.15.181:0/1364991307 >> 172.21.15.181:0/1364991307 conn(0x7fcb1c0fc500 msgr2=0x7fcb1c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:21.026 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.035+0000 7fcb24301700 1 -- 172.21.15.181:0/1364991307 shutdown_connections 2024-04-27T01:10:21.026 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.035+0000 7fcb24301700 1 -- 172.21.15.181:0/1364991307 wait complete. 2024-04-27T01:10:21.026 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.036+0000 7fcb24301700 1 Processor -- start 2024-04-27T01:10:21.026 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.036+0000 7fcb24301700 1 -- start start 2024-04-27T01:10:21.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.036+0000 7fcb24301700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcb1c198350 con 0x7fcb1c102b50 2024-04-27T01:10:21.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.036+0000 7fcb24301700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcb1c1a3b40 con 0x7fcb1c106660 2024-04-27T01:10:21.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.036+0000 7fcb24301700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcb1c1a4d20 con 0x7fcb1c10a170 2024-04-27T01:10:21.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.036+0000 7fcb2189c700 1 --1- >> v1:172.21.15.181:6789/0 conn(0x7fcb1c106660 0x7fcb1c19dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.181:6789/0 says I am v1:172.21.15.181:59424/0 (socket says 172.21.15.181:59424) 2024-04-27T01:10:21.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.036+0000 7fcb2189c700 1 -- 172.21.15.181:0/1686896902 learned_addr learned my addr 172.21.15.181:0/1686896902 (peer_addr_for_me v1:172.21.15.181:0/0) 2024-04-27T01:10:21.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.036+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684468056 0 0) 0x7fcb1c1a3b40 con 0x7fcb1c106660 2024-04-27T01:10:21.027 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.036+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb080035b0 con 0x7fcb1c106660 2024-04-27T01:10:21.028 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2807197976 0 0) 0x7fcb1c198350 con 0x7fcb1c102b50 2024-04-27T01:10:21.028 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb1c1a3b40 con 0x7fcb1c102b50 2024-04-27T01:10:21.028 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2418121427 0 0) 0x7fcb080035b0 con 0x7fcb1c106660 2024-04-27T01:10:21.028 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb1c198350 con 0x7fcb1c106660 2024-04-27T01:10:21.028 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fcb0c001930 con 0x7fcb1c106660 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143528274 0 0) 0x7fcb1c1a4d20 con 0x7fcb1c10a170 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb080035b0 con 0x7fcb1c10a170 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3911565628 0 0) 0x7fcb1c198350 con 0x7fcb1c106660 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 >> v1:172.21.15.162:6790/0 conn(0x7fcb1c10a170 legacy=0x7fcb1c1a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 >> v1:172.21.15.162:6789/0 conn(0x7fcb1c102b50 legacy=0x7fcb1c197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c1a5f40 con 0x7fcb1c106660 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb24301700 1 -- 172.21.15.181:0/1686896902 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1c1a2b90 con 0x7fcb1c106660 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb24301700 1 -- 172.21.15.181:0/1686896902 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcb1c1a30d0 con 0x7fcb1c106660 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fcb0c001d00 con 0x7fcb1c106660 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.037+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fcb0c005e10 con 0x7fcb1c106660 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.038+0000 7fcb24301700 1 -- 172.21.15.181:0/1686896902 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb00005150 con 0x7fcb1c106660 2024-04-27T01:10:21.029 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.038+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 16) ==== 92133+0+0 (unknown 3105418878 0 0) 0x7fcb0c0039d0 con 0x7fcb1c106660 2024-04-27T01:10:21.030 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.039+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(70..70 src has 1..70) ==== 5942+0+0 (unknown 665183937 0 0) 0x7fcb0c0896e0 con 0x7fcb1c106660 2024-04-27T01:10:21.033 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.043+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fcb0c056dc0 con 0x7fcb1c106660 2024-04-27T01:10:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-27T01:10:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-27T01:10:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:21 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-27T01:10:21.320 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.330+0000 7fcb24301700 1 -- 172.21.15.181:0/1686896902 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fcb00005440 con 0x7fcb1c106660 2024-04-27T01:10:21.324 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.335+0000 7fcb137fe700 1 -- 172.21.15.181:0/1686896902 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) ==== 170+0+59 (unknown 3508017507 0 1380507538) 0x7fcb0c0566c0 con 0x7fcb1c106660 2024-04-27T01:10:21.324 INFO:teuthology.orchestra.run.smithi181.stdout:[client.1] 2024-04-27T01:10:21.325 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQB9UCxmPovJExAAXjfV26rAHEaJ/UO7ArBA+A== 2024-04-27T01:10:21.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.337+0000 7fcb24301700 1 -- 172.21.15.181:0/1686896902 >> v1:172.21.15.162:6800/2166195007 conn(0x7fcb0806eb40 legacy=0x7fcb08071000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:21.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.337+0000 7fcb24301700 1 -- 172.21.15.181:0/1686896902 >> v1:172.21.15.181:6789/0 conn(0x7fcb1c106660 legacy=0x7fcb1c19dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:21.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.337+0000 7fcb24301700 1 -- 172.21.15.181:0/1686896902 shutdown_connections 2024-04-27T01:10:21.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.337+0000 7fcb24301700 1 -- 172.21.15.181:0/1686896902 >> 172.21.15.181:0/1686896902 conn(0x7fcb1c0fc500 msgr2=0x7fcb1c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:21.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.337+0000 7fcb24301700 1 -- 172.21.15.181:0/1686896902 shutdown_connections 2024-04-27T01:10:21.327 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-27T01:10:21.337+0000 7fcb24301700 1 -- 172.21.15.181:0/1686896902 wait complete. 2024-04-27T01:10:21.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-27T01:10:21.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-27T01:10:21.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:21 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-27T01:10:21.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-27T01:10:21.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-27T01:10:21.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:21 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-04-27T01:10:22.199 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-27T01:10:22.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-27T01:10:22.200 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-27T01:10:22.255 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-27T01:10:22.255 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-27T01:10:22.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-27T01:10:22.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:22 smithi181 ceph-mon[30237]: pgmap v210: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-27T01:10:22.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:22 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-27T01:10:22.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:22 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-27T01:10:22.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:22 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-04-27T01:10:22.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:22 smithi181 ceph-mon[30237]: osdmap e71: 8 total, 8 up, 8 in 2024-04-27T01:10:22.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.181:0/1686896902' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:22.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:22 smithi181 ceph-mon[30237]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:22.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:22 smithi181 ceph-mon[30237]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T01:10:22.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[26714]: pgmap v210: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-27T01:10:22.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-27T01:10:22.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-27T01:10:22.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-04-27T01:10:22.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[26714]: osdmap e71: 8 total, 8 up, 8 in 2024-04-27T01:10:22.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.181:0/1686896902' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:22.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[26714]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:22.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[26714]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T01:10:22.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[37680]: pgmap v210: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-04-27T01:10:22.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-27T01:10:22.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-27T01:10:22.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-04-27T01:10:22.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[37680]: osdmap e71: 8 total, 8 up, 8 in 2024-04-27T01:10:22.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.181:0/1686896902' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:22.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[37680]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-27T01:10:22.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:22 smithi162 ceph-mon[37680]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-27T01:10:22.502 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:23.011 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:10:22 smithi181 systemd[1]: Started Ceph prometheus.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:10:23.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-mon[37680]: osdmap e72: 8 total, 8 up, 8 in 2024-04-27T01:10:23.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:23.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:23.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:23.282 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-27T01:10:23.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-mon[26714]: osdmap e72: 8 total, 8 up, 8 in 2024-04-27T01:10:23.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:23.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:23.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:23.283 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-27T01:10:23.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:23 smithi181 ceph-mon[30237]: osdmap e72: 8 total, 8 up, 8 in 2024-04-27T01:10:23.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:23.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:23.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' 2024-04-27T01:10:23.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:23 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-27T01:10:24.218 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-mon[26714]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-27T01:10:24.218 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-mon[26714]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-27T01:10:24.218 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-mon[26714]: mgrmap e17: y(active, since 5m), standbys: x 2024-04-27T01:10:24.218 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ignoring --setuser ceph since I am not root 2024-04-27T01:10:24.218 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:23 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ignoring --setgroup ceph since I am not root 2024-04-27T01:10:24.218 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:24.087+0000 7f9030f7b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-27T01:10:24.219 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-mon[37680]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-27T01:10:24.219 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-mon[37680]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-27T01:10:24.219 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-mon[37680]: mgrmap e17: y(active, since 5m), standbys: x 2024-04-27T01:10:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.227+0000 7f299547c700 1 -- 172.21.15.162:0/485604671 >> v1:172.21.15.162:6790/0 conn(0x7f299010ea80 legacy=0x7f2990110f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.227+0000 7f299547c700 1 -- 172.21.15.162:0/485604671 shutdown_connections 2024-04-27T01:10:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.227+0000 7f299547c700 1 -- 172.21.15.162:0/485604671 >> 172.21.15.162:0/485604671 conn(0x7f29900fdb40 msgr2=0x7f29900fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.227+0000 7f299547c700 1 -- 172.21.15.162:0/485604671 shutdown_connections 2024-04-27T01:10:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.227+0000 7f299547c700 1 -- 172.21.15.162:0/485604671 wait complete. 2024-04-27T01:10:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.228+0000 7f299547c700 1 Processor -- start 2024-04-27T01:10:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.228+0000 7f299547c700 1 -- start start 2024-04-27T01:10:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.228+0000 7f299547c700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29901a7050 con 0x7f2990102250 2024-04-27T01:10:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.228+0000 7f299547c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29901a8270 con 0x7f299010aee0 2024-04-27T01:10:24.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.228+0000 7f299547c700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29901a94b0 con 0x7f299010ea80 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.228+0000 7f298effd700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f2990102250 0x7f29901074f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:51768/0 (socket says 172.21.15.162:51768) 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.228+0000 7f298effd700 1 -- 172.21.15.162:0/4026116257 learned_addr learned my addr 172.21.15.162:0/4026116257 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2596740201 0 0) 0x7f29901a8270 con 0x7f299010aee0 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29780031f0 con 0x7f299010aee0 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1168872737 0 0) 0x7f29901a94b0 con 0x7f299010ea80 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29901a8270 con 0x7f299010ea80 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3237181875 0 0) 0x7f29901a7050 con 0x7f2990102250 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29901a94b0 con 0x7f2990102250 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 465423208 0 0) 0x7f29901a8270 con 0x7f299010ea80 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29901a7050 con 0x7f299010ea80 2024-04-27T01:10:24.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 132040747 0 0) 0x7f29780031f0 con 0x7f299010aee0 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29901a8270 con 0x7f299010aee0 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2988001d00 con 0x7f299010ea80 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1598264319 0 0) 0x7f29901a94b0 con 0x7f2990102250 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29780031f0 con 0x7f2990102250 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2980001830 con 0x7f299010aee0 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2984001890 con 0x7f2990102250 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 440845572 0 0) 0x7f29901a7050 con 0x7f299010ea80 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 >> v1:172.21.15.181:6789/0 conn(0x7f299010aee0 legacy=0x7f29901a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 >> v1:172.21.15.162:6789/0 conn(0x7f2990102250 legacy=0x7f29901074f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29901aa6f0 con 0x7f299010ea80 2024-04-27T01:10:24.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f299547c700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29901a84a0 con 0x7f299010ea80 2024-04-27T01:10:24.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f299547c700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f29901a8960 con 0x7f299010ea80 2024-04-27T01:10:24.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f2988001ff0 con 0x7f299010ea80 2024-04-27T01:10:24.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.229+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2988004f70 con 0x7f299010ea80 2024-04-27T01:10:24.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.231+0000 7f299547c700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f297c005150 con 0x7f299010ea80 2024-04-27T01:10:24.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.231+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 17) ==== 92147+0+0 (unknown 1649489690 0 0) 0x7f298801b990 con 0x7f299010ea80 2024-04-27T01:10:24.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.232+0000 7f298effd700 1 -- 172.21.15.162:0/4026116257 >> v1:172.21.15.162:6800/2166195007 conn(0x7f297806f000 legacy=0x7f29780714c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/2166195007 2024-04-27T01:10:24.224 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.232+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(72..72 src has 1..72) ==== 6001+0+0 (unknown 486092737 0 0) 0x7f2988088620 con 0x7f299010ea80 2024-04-27T01:10:24.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.237+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f2988055d80 con 0x7f299010ea80 2024-04-27T01:10:24.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:24 smithi181 ceph-mon[30237]: pgmap v213: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-27T01:10:24.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:24 smithi181 ceph-mon[30237]: from='mgr.14152 v1:172.21.15.162:0/1018600444' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-27T01:10:24.248 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:24 smithi181 ceph-mon[30237]: mgrmap e17: y(active, since 5m), standbys: x 2024-04-27T01:10:24.249 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: ignoring --setuser ceph since I am not root 2024-04-27T01:10:24.249 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: ignoring --setgroup ceph since I am not root 2024-04-27T01:10:24.249 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:24.075+0000 7f85e1202200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-27T01:10:24.249 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:24.258+0000 7f85e1202200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:10:24.423 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.433+0000 7f298effd700 1 -- 172.21.15.162:0/4026116257 >> v1:172.21.15.162:6800/2166195007 conn(0x7f297806f000 legacy=0x7f29780714c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/2166195007 2024-04-27T01:10:24.490 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.499+0000 7f299547c700 1 -- 172.21.15.162:0/4026116257 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f297c005440 con 0x7f299010ea80 2024-04-27T01:10:24.490 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:24.279+0000 7f9030f7b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:10:24.490 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:24.385+0000 7f9030f7b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-27T01:10:24.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.503+0000 7f2977fff700 1 -- 172.21.15.162:0/4026116257 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+175962 (unknown 831476310 0 4289885401) 0x7f2988001380 con 0x7f299010ea80 2024-04-27T01:10:24.496 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:10:24.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.506+0000 7f299547c700 1 -- 172.21.15.162:0/4026116257 >> v1:172.21.15.162:6800/2166195007 conn(0x7f297806f000 legacy=0x7f29780714c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:10:24.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.506+0000 7f299547c700 1 -- 172.21.15.162:0/4026116257 >> v1:172.21.15.162:6790/0 conn(0x7f299010ea80 legacy=0x7f29901a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:24.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.506+0000 7f299547c700 1 -- 172.21.15.162:0/4026116257 shutdown_connections 2024-04-27T01:10:24.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.506+0000 7f299547c700 1 -- 172.21.15.162:0/4026116257 >> 172.21.15.162:0/4026116257 conn(0x7f29900fdb40 msgr2=0x7f2990111bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:24.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.506+0000 7f299547c700 1 -- 172.21.15.162:0/4026116257 shutdown_connections 2024-04-27T01:10:24.498 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:24.506+0000 7f299547c700 1 -- 172.21.15.162:0/4026116257 wait complete. 2024-04-27T01:10:24.511 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:24.360+0000 7f85e1202200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-27T01:10:24.782 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:24.704+0000 7f9030f7b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-27T01:10:25.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:24.662+0000 7f85e1202200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-27T01:10:25.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:24.744+0000 7f85e1202200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-27T01:10:25.077 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:24 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:24.792+0000 7f9030f7b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-27T01:10:25.137 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6800","nonce":2166195007}]},"active_addr":"172.21.15.162:6800/2166195007","active_change":"2024-04-27T01:05:20.958477+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.162:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":96615509}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":2558377069}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":1081933236}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":219736518}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":3003976485}]}]} 2024-04-27T01:10:25.141 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-27T01:10:25.141 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-27T01:10:25.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T01:10:25.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4026116257' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T01:10:25.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4026116257' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T01:10:25.422 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:25.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4026116257' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-27T01:10:26.239 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:25 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:25.907+0000 7f85e1202200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-27T01:10:26.239 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:26.078+0000 7f85e1202200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-27T01:10:26.339 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:26 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:26.053+0000 7f9030f7b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-27T01:10:26.339 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:26 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:26.227+0000 7f9030f7b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-27T01:10:26.511 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:26.249+0000 7f85e1202200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-27T01:10:26.690 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:26 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:26.400+0000 7f9030f7b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-27T01:10:26.823 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:26.538+0000 7f85e1202200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-27T01:10:26.824 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:26.631+0000 7f85e1202200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-27T01:10:26.996 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:26 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:26.700+0000 7f9030f7b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-27T01:10:26.996 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:26 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:26.796+0000 7f9030f7b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-27T01:10:27.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.138+0000 7f039f96c700 1 -- 172.21.15.162:0/237683834 >> v1:172.21.15.162:6790/0 conn(0x7f0398109f00 legacy=0x7f039810c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:27.128 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.138+0000 7f039f96c700 1 -- 172.21.15.162:0/237683834 shutdown_connections 2024-04-27T01:10:27.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.138+0000 7f039f96c700 1 -- 172.21.15.162:0/237683834 >> 172.21.15.162:0/237683834 conn(0x7f03980fca10 msgr2=0x7f03980fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:27.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.138+0000 7f039f96c700 1 -- 172.21.15.162:0/237683834 shutdown_connections 2024-04-27T01:10:27.129 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.138+0000 7f039f96c700 1 -- 172.21.15.162:0/237683834 wait complete. 2024-04-27T01:10:27.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f039f96c700 1 Processor -- start 2024-04-27T01:10:27.131 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f039f96c700 1 -- start start 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f039f96c700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0398106b10 con 0x7f039810daa0 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f039f96c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03981a8680 con 0x7f0398101160 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f039f96c700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f03981a98a0 con 0x7f0398109f00 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f039cf07700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f0398109f00 0x7f0398104ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:39500/0 (socket says 172.21.15.162:39500) 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f039cf07700 1 -- 172.21.15.162:0/2560002764 learned_addr learned my addr 172.21.15.162:0/2560002764 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f039df09700 1 --1- 172.21.15.162:0/2560002764 >> v1:172.21.15.162:6789/0 conn(0x7f039810daa0 0x7f03981a5f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47386/0 (socket says 172.21.15.162:47386) 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 836747861 0 0) 0x7f0398106b10 con 0x7f039810daa0 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03840035b0 con 0x7f039810daa0 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1167294005 0 0) 0x7f03981a8680 con 0x7f0398101160 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0398106b10 con 0x7f0398101160 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 986506869 0 0) 0x7f03981a98a0 con 0x7f0398109f00 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.139+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03981a8680 con 0x7f0398109f00 2024-04-27T01:10:27.132 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1688646542 0 0) 0x7f03840035b0 con 0x7f039810daa0 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03981a98a0 con 0x7f039810daa0 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f0390001c30 con 0x7f039810daa0 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2279011784 0 0) 0x7f0398106b10 con 0x7f0398101160 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f03840035b0 con 0x7f0398101160 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1223004102 0 0) 0x7f03981a8680 con 0x7f0398109f00 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0398106b10 con 0x7f0398109f00 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f03940017f0 con 0x7f0398101160 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f0388001430 con 0x7f0398109f00 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 917981711 0 0) 0x7f03981a98a0 con 0x7f039810daa0 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 >> v1:172.21.15.162:6790/0 conn(0x7f0398109f00 legacy=0x7f0398104ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 >> v1:172.21.15.181:6789/0 conn(0x7f0398101160 legacy=0x7f03981048e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03981aaac0 con 0x7f039810daa0 2024-04-27T01:10:27.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f039f96c700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f03981a88b0 con 0x7f039810daa0 2024-04-27T01:10:27.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f039f96c700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f03981a8d70 con 0x7f039810daa0 2024-04-27T01:10:27.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f0390003eb0 con 0x7f039810daa0 2024-04-27T01:10:27.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.140+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f0390005f80 con 0x7f039810daa0 2024-04-27T01:10:27.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.141+0000 7f039f96c700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f037c005120 con 0x7f039810daa0 2024-04-27T01:10:27.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.141+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 17) ==== 92147+0+0 (unknown 1649489690 0 0) 0x7f0390003a20 con 0x7f039810daa0 2024-04-27T01:10:27.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.142+0000 7f039d708700 1 -- 172.21.15.162:0/2560002764 >> v1:172.21.15.162:6800/2166195007 conn(0x7f038406f000 legacy=0x7f03840714c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/2166195007 2024-04-27T01:10:27.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.142+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(72..72 src has 1..72) ==== 6001+0+0 (unknown 486092737 0 0) 0x7f0390089840 con 0x7f039810daa0 2024-04-27T01:10:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.147+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f0390056f20 con 0x7f039810daa0 2024-04-27T01:10:27.261 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:26.833+0000 7f85e1202200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-27T01:10:27.261 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:27.025+0000 7f85e1202200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:10:27.333 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.343+0000 7f039d708700 1 -- 172.21.15.162:0/2560002764 >> v1:172.21.15.162:6800/2166195007 conn(0x7f038406f000 legacy=0x7f03840714c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/2166195007 2024-04-27T01:10:27.333 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:27 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:27.006+0000 7f9030f7b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-27T01:10:27.333 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:27 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:27.212+0000 7f9030f7b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-27T01:10:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.353+0000 7f039f96c700 1 -- 172.21.15.162:0/2560002764 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f037c005410 con 0x7f039810daa0 2024-04-27T01:10:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.355+0000 7f038e7fc700 1 -- 172.21.15.162:0/2560002764 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+20700 (unknown 1686360691 0 2931098896) 0x7f0390056820 con 0x7f039810daa0 2024-04-27T01:10:27.345 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:10:27.346 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":72,"fsid":"ea378f3c-0431-11ef-bc93-c7b262605968","created":"2024-04-27T01:04:19.059661+0000","modified":"2024-04-27T01:10:22.071935+0000","last_up_change":"2024-04-27T01:09:38.516534+0000","last_in_change":"2024-04-27T01:09:20.973588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T01:07:37.280446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-04-27T01:09:49.767587+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-04-27T01:09:50.358308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-04-27T01:09:52.278721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-27T01:09:54.224356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-27T01:09:56.283202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8f1e299d-4d29-4333-b632-249ff9f12d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":2041814805}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":2041814805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":2041814805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":2041814805}]},"public_addr":"172.21.15.162:6801/2041814805","cluster_addr":"172.21.15.162:6802/2041814805","heartbeat_back_addr":"172.21.15.162:6804/2041814805","heartbeat_front_addr":"172.21.15.162:6803/2041814805","state":["exists","up"]},{"osd":1,"uuid":"9013bcef-adce-49f2-952c-f3f208bf4e96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":680367434}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":680367434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":680367434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":680367434}]},"public_addr":"172.21.15.162:6805/680367434","cluster_addr":"172.21.15.162:6806/680367434","heartbeat_back_addr":"172.21.15.162:6808/680367434","heartbeat_front_addr":"172.21.15.162:6807/680367434","state":["exists","up"]},{"osd":2,"uuid":"f2212181-797c-4366-b087-2522d5db9e95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":2147722624}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":2147722624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":2147722624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":2147722624}]},"public_addr":"172.21.15.162:6809/2147722624","cluster_addr":"172.21.15.162:6810/2147722624","heartbeat_back_addr":"172.21.15.162:6812/2147722624","heartbeat_front_addr":"172.21.15.162:6811/2147722624","state":["exists","up"]},{"osd":3,"uuid":"21e98e27-d849-4b8c-9db9-98e738395fd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":2875926639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":2875926639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6816","nonce":2875926639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":2875926639}]},"public_addr":"172.21.15.162:6813/2875926639","cluster_addr":"172.21.15.162:6814/2875926639","heartbeat_back_addr":"172.21.15.162:6816/2875926639","heartbeat_front_addr":"172.21.15.162:6815/2875926639","state":["exists","up"]},{"osd":4,"uuid":"ad7cfc95-49e8-440a-8e15-55e6900ae122","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6800","nonce":1869417285}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6801","nonce":1869417285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6803","nonce":1869417285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6802","nonce":1869417285}]},"public_addr":"172.21.15.181:6800/1869417285","cluster_addr":"172.21.15.181:6801/1869417285","heartbeat_back_addr":"172.21.15.181:6803/1869417285","heartbeat_front_addr":"172.21.15.181:6802/1869417285","state":["exists","up"]},{"osd":5,"uuid":"46341ac9-3fc7-49c0-bc92-03ba76da4ec7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6804","nonce":3172409639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6805","nonce":3172409639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6807","nonce":3172409639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6806","nonce":3172409639}]},"public_addr":"172.21.15.181:6804/3172409639","cluster_addr":"172.21.15.181:6805/3172409639","heartbeat_back_addr":"172.21.15.181:6807/3172409639","heartbeat_front_addr":"172.21.15.181:6806/3172409639","state":["exists","up"]},{"osd":6,"uuid":"832670c0-ca6f-484e-8719-3cf680f03fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6808","nonce":30247032}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6809","nonce":30247032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6811","nonce":30247032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6810","nonce":30247032}]},"public_addr":"172.21.15.181:6808/30247032","cluster_addr":"172.21.15.181:6809/30247032","heartbeat_back_addr":"172.21.15.181:6811/30247032","heartbeat_front_addr":"172.21.15.181:6810/30247032","state":["exists","up"]},{"osd":7,"uuid":"63a81b98-465c-4a37-a666-90133708abc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6812","nonce":1269032985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6813","nonce":1269032985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6815","nonce":1269032985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6814","nonce":1269032985}]},"public_addr":"172.21.15.181:6812/1269032985","cluster_addr":"172.21.15.181:6813/1269032985","heartbeat_back_addr":"172.21.15.181:6815/1269032985","heartbeat_front_addr":"172.21.15.181:6814/1269032985","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:06:41.989409+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:07:06.920751+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:07:32.636741+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:07:58.313419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:08:21.343521+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:08:46.671407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:09:12.035903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:09:36.866889+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:0/165265347":"2024-04-28T01:05:20.958366+0000","172.21.15.162:0/1636118121":"2024-04-28T01:04:54.155544+0000","172.21.15.162:0/837853490":"2024-04-28T01:04:54.155544+0000","172.21.15.162:0/3941236233":"2024-04-28T01:05:20.958366+0000","172.21.15.162:0/104247678":"2024-04-28T01:04:54.155544+0000","172.21.15.162:0/2952006630":"2024-04-28T01:05:20.958366+0000","172.21.15.162:6800/629135069":"2024-04-28T01:04:54.155544+0000","172.21.15.162:6800/106205076":"2024-04-28T01:05:20.958366+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T01:10:27.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.357+0000 7f039f96c700 1 -- 172.21.15.162:0/2560002764 >> v1:172.21.15.162:6800/2166195007 conn(0x7f038406f000 legacy=0x7f03840714c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:10:27.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.357+0000 7f039f96c700 1 -- 172.21.15.162:0/2560002764 >> v1:172.21.15.162:6789/0 conn(0x7f039810daa0 legacy=0x7f03981a5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:27.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.357+0000 7f039f96c700 1 -- 172.21.15.162:0/2560002764 shutdown_connections 2024-04-27T01:10:27.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.357+0000 7f039f96c700 1 -- 172.21.15.162:0/2560002764 >> 172.21.15.162:0/2560002764 conn(0x7f03980fca10 msgr2=0x7f039806a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:27.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.357+0000 7f039f96c700 1 -- 172.21.15.162:0/2560002764 shutdown_connections 2024-04-27T01:10:27.348 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:27.358+0000 7f039f96c700 1 -- 172.21.15.162:0/2560002764 wait complete. 2024-04-27T01:10:27.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2560002764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:10:27.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2560002764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:10:27.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2560002764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:10:27.761 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:27.630+0000 7f85e1202200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-27T01:10:27.941 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:27 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:27.832+0000 7f9030f7b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-27T01:10:27.970 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-27T01:10:27.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-27T01:10:28.214 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:28.340 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:28.053+0000 7f85e1202200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-27T01:10:28.340 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:28.148+0000 7f85e1202200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-27T01:10:28.340 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:28.350+0000 7f85e1202200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-27T01:10:28.589 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:28 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:28.294+0000 7f9030f7b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-27T01:10:28.589 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:28 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:28.395+0000 7f9030f7b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-27T01:10:29.089 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:28 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:28.607+0000 7f9030f7b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-27T01:10:29.203 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:28.878+0000 7f85e1202200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-27T01:10:29.204 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:28.970+0000 7f85e1202200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-27T01:10:29.475 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:29.141+0000 7f9030f7b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-27T01:10:29.475 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:29.235+0000 7f9030f7b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-27T01:10:29.511 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:29.213+0000 7f85e1202200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-27T01:10:29.511 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:29.346+0000 7f85e1202200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-27T01:10:29.511 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:29.443+0000 7f85e1202200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-27T01:10:29.833 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:29.483+0000 7f9030f7b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-27T01:10:29.833 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:29.624+0000 7f9030f7b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-27T01:10:29.834 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:29.723+0000 7f9030f7b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-27T01:10:29.834 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:29.842+0000 7f9030f7b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-27T01:10:30.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:29.559+0000 7f85e1202200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-27T01:10:30.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:29.652+0000 7f85e1202200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-27T01:10:30.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.080+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2515939152 >> v1:172.21.15.162:6790/0 conn(0x7f1edc1031c0 legacy=0x7f1edc1035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:30.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.080+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2515939152 shutdown_connections 2024-04-27T01:10:30.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.080+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2515939152 >> 172.21.15.162:0/2515939152 conn(0x7f1edc0fca10 msgr2=0x7f1edc0fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:30.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.080+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2515939152 shutdown_connections 2024-04-27T01:10:30.072 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.080+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2515939152 wait complete. 2024-04-27T01:10:30.074 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.081+0000 7f1ee2b0f700 1 Processor -- start 2024-04-27T01:10:30.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.081+0000 7f1ee2b0f700 1 -- start start 2024-04-27T01:10:30.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ee2b0f700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1edc1a7210 con 0x7f1edc1031c0 2024-04-27T01:10:30.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ee2b0f700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1edc1a83f0 con 0x7f1edc10b0b0 2024-04-27T01:10:30.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ee2b0f700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1edc1a9630 con 0x7f1edc10ec50 2024-04-27T01:10:30.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ee10ac700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f1edc10ec50 0x7f1edc1a58a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:39514/0 (socket says 172.21.15.162:39514) 2024-04-27T01:10:30.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ee10ac700 1 -- 172.21.15.162:0/2273858840 learned_addr learned my addr 172.21.15.162:0/2273858840 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:30.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ee08ab700 1 --1- 172.21.15.162:0/2273858840 >> v1:172.21.15.162:6789/0 conn(0x7f1edc1031c0 0x7f1edc10a840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47406/0 (socket says 172.21.15.162:47406) 2024-04-27T01:10:30.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3271183281 0 0) 0x7f1edc1a83f0 con 0x7f1edc10b0b0 2024-04-27T01:10:30.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ecc0031f0 con 0x7f1edc10b0b0 2024-04-27T01:10:30.075 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899573843 0 0) 0x7f1edc1a7210 con 0x7f1edc1031c0 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1edc1a83f0 con 0x7f1edc1031c0 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3588329618 0 0) 0x7f1edc1a9630 con 0x7f1edc10ec50 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1edc1a7210 con 0x7f1edc10ec50 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3822808596 0 0) 0x7f1ecc0031f0 con 0x7f1edc10b0b0 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1edc1a9630 con 0x7f1edc10b0b0 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1ec8001960 con 0x7f1edc10b0b0 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 329478138 0 0) 0x7f1edc1a83f0 con 0x7f1edc1031c0 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ecc0031f0 con 0x7f1edc1031c0 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2373441988 0 0) 0x7f1edc1a7210 con 0x7f1edc10ec50 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1edc1a83f0 con 0x7f1edc10ec50 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1ed0004470 con 0x7f1edc1031c0 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.082+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1ed4001bf0 con 0x7f1edc10ec50 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.083+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 980168678 0 0) 0x7f1edc1a9630 con 0x7f1edc10b0b0 2024-04-27T01:10:30.076 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.083+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 >> v1:172.21.15.162:6790/0 conn(0x7f1edc10ec50 legacy=0x7f1edc1a58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.083+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 >> v1:172.21.15.162:6789/0 conn(0x7f1edc1031c0 legacy=0x7f1edc10a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.083+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc1aa870 con 0x7f1edc10b0b0 2024-04-27T01:10:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.083+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1edc1a8620 con 0x7f1edc10b0b0 2024-04-27T01:10:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.083+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f1ec8003b10 con 0x7f1edc10b0b0 2024-04-27T01:10:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.083+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1ec8004980 con 0x7f1edc10b0b0 2024-04-27T01:10:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.083+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1edc1a8ae0 con 0x7f1edc10b0b0 2024-04-27T01:10:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.084+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1edc06ad50 con 0x7f1edc10b0b0 2024-04-27T01:10:30.077 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.084+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 17) ==== 92147+0+0 (unknown 1649489690 0 0) 0x7f1ec8003680 con 0x7f1edc10b0b0 2024-04-27T01:10:30.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.085+0000 7f1ee08ab700 1 -- 172.21.15.162:0/2273858840 >> v1:172.21.15.162:6800/2166195007 conn(0x7f1ecc06efa0 legacy=0x7f1ecc071460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/2166195007 2024-04-27T01:10:30.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.085+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(72..72 src has 1..72) ==== 6001+0+0 (unknown 486092737 0 0) 0x7f1ec8088260 con 0x7f1edc10b0b0 2024-04-27T01:10:30.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.090+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f1ec8055900 con 0x7f1edc10b0b0 2024-04-27T01:10:30.089 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:29.939+0000 7f9030f7b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-27T01:10:30.277 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.286+0000 7f1ee08ab700 1 -- 172.21.15.162:0/2273858840 >> v1:172.21.15.162:6800/2166195007 conn(0x7f1ecc06efa0 legacy=0x7f1ecc071460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.162:6800/2166195007 2024-04-27T01:10:30.286 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.295+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2273858840 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1edc068b50 con 0x7f1edc10b0b0 2024-04-27T01:10:30.288 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.297+0000 7f1ed9ffb700 1 -- 172.21.15.162:0/2273858840 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+20700 (unknown 1686360691 0 2931098896) 0x7f1ec8055200 con 0x7f1edc10b0b0 2024-04-27T01:10:30.288 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:10:30.288 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":72,"fsid":"ea378f3c-0431-11ef-bc93-c7b262605968","created":"2024-04-27T01:04:19.059661+0000","modified":"2024-04-27T01:10:22.071935+0000","last_up_change":"2024-04-27T01:09:38.516534+0000","last_in_change":"2024-04-27T01:09:20.973588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T01:07:37.280446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-04-27T01:09:49.767587+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-04-27T01:09:50.358308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-04-27T01:09:52.278721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-27T01:09:54.224356+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-27T01:09:56.283202+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8f1e299d-4d29-4333-b632-249ff9f12d5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6801","nonce":2041814805}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6802","nonce":2041814805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6804","nonce":2041814805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6803","nonce":2041814805}]},"public_addr":"172.21.15.162:6801/2041814805","cluster_addr":"172.21.15.162:6802/2041814805","heartbeat_back_addr":"172.21.15.162:6804/2041814805","heartbeat_front_addr":"172.21.15.162:6803/2041814805","state":["exists","up"]},{"osd":1,"uuid":"9013bcef-adce-49f2-952c-f3f208bf4e96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6805","nonce":680367434}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6806","nonce":680367434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6808","nonce":680367434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6807","nonce":680367434}]},"public_addr":"172.21.15.162:6805/680367434","cluster_addr":"172.21.15.162:6806/680367434","heartbeat_back_addr":"172.21.15.162:6808/680367434","heartbeat_front_addr":"172.21.15.162:6807/680367434","state":["exists","up"]},{"osd":2,"uuid":"f2212181-797c-4366-b087-2522d5db9e95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6809","nonce":2147722624}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6810","nonce":2147722624}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6812","nonce":2147722624}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6811","nonce":2147722624}]},"public_addr":"172.21.15.162:6809/2147722624","cluster_addr":"172.21.15.162:6810/2147722624","heartbeat_back_addr":"172.21.15.162:6812/2147722624","heartbeat_front_addr":"172.21.15.162:6811/2147722624","state":["exists","up"]},{"osd":3,"uuid":"21e98e27-d849-4b8c-9db9-98e738395fd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6813","nonce":2875926639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6814","nonce":2875926639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6816","nonce":2875926639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.162:6815","nonce":2875926639}]},"public_addr":"172.21.15.162:6813/2875926639","cluster_addr":"172.21.15.162:6814/2875926639","heartbeat_back_addr":"172.21.15.162:6816/2875926639","heartbeat_front_addr":"172.21.15.162:6815/2875926639","state":["exists","up"]},{"osd":4,"uuid":"ad7cfc95-49e8-440a-8e15-55e6900ae122","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6800","nonce":1869417285}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6801","nonce":1869417285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6803","nonce":1869417285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6802","nonce":1869417285}]},"public_addr":"172.21.15.181:6800/1869417285","cluster_addr":"172.21.15.181:6801/1869417285","heartbeat_back_addr":"172.21.15.181:6803/1869417285","heartbeat_front_addr":"172.21.15.181:6802/1869417285","state":["exists","up"]},{"osd":5,"uuid":"46341ac9-3fc7-49c0-bc92-03ba76da4ec7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6804","nonce":3172409639}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6805","nonce":3172409639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6807","nonce":3172409639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6806","nonce":3172409639}]},"public_addr":"172.21.15.181:6804/3172409639","cluster_addr":"172.21.15.181:6805/3172409639","heartbeat_back_addr":"172.21.15.181:6807/3172409639","heartbeat_front_addr":"172.21.15.181:6806/3172409639","state":["exists","up"]},{"osd":6,"uuid":"832670c0-ca6f-484e-8719-3cf680f03fb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6808","nonce":30247032}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6809","nonce":30247032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6811","nonce":30247032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6810","nonce":30247032}]},"public_addr":"172.21.15.181:6808/30247032","cluster_addr":"172.21.15.181:6809/30247032","heartbeat_back_addr":"172.21.15.181:6811/30247032","heartbeat_front_addr":"172.21.15.181:6810/30247032","state":["exists","up"]},{"osd":7,"uuid":"63a81b98-465c-4a37-a666-90133708abc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6812","nonce":1269032985}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6813","nonce":1269032985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6815","nonce":1269032985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.181:6814","nonce":1269032985}]},"public_addr":"172.21.15.181:6812/1269032985","cluster_addr":"172.21.15.181:6813/1269032985","heartbeat_back_addr":"172.21.15.181:6815/1269032985","heartbeat_front_addr":"172.21.15.181:6814/1269032985","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:06:41.989409+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:07:06.920751+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:07:32.636741+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:07:58.313419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:08:21.343521+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:08:46.671407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:09:12.035903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T01:09:36.866889+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:0/165265347":"2024-04-28T01:05:20.958366+0000","172.21.15.162:0/1636118121":"2024-04-28T01:04:54.155544+0000","172.21.15.162:0/837853490":"2024-04-28T01:04:54.155544+0000","172.21.15.162:0/3941236233":"2024-04-28T01:05:20.958366+0000","172.21.15.162:0/104247678":"2024-04-28T01:04:54.155544+0000","172.21.15.162:0/2952006630":"2024-04-28T01:05:20.958366+0000","172.21.15.162:6800/629135069":"2024-04-28T01:04:54.155544+0000","172.21.15.162:6800/106205076":"2024-04-28T01:05:20.958366+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T01:10:30.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.299+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2273858840 >> v1:172.21.15.162:6800/2166195007 conn(0x7f1ecc06efa0 legacy=0x7f1ecc071460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-27T01:10:30.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.299+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2273858840 >> v1:172.21.15.181:6789/0 conn(0x7f1edc10b0b0 legacy=0x7f1edc1a2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:30.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.299+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2273858840 shutdown_connections 2024-04-27T01:10:30.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.299+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2273858840 >> 172.21.15.162:0/2273858840 conn(0x7f1edc0fca10 msgr2=0x7f1edc0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:30.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.299+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2273858840 shutdown_connections 2024-04-27T01:10:30.290 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:30.299+0000 7f1ee2b0f700 1 -- 172.21.15.162:0/2273858840 wait complete. 2024-04-27T01:10:30.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2273858840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:10:30.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2273858840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:10:30.680 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2273858840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T01:10:30.681 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:30.472+0000 7f85e1202200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-27T01:10:30.965 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:30 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:30.787+0000 7f9030f7b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-27T01:10:30.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-27T01:10:30.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-27T01:10:30.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-27T01:10:30.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-27T01:10:30.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-27T01:10:30.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-27T01:10:30.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-27T01:10:30.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-27T01:10:31.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:30.690+0000 7f85e1202200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-27T01:10:31.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: [27/Apr/2024:01:10:30] ENGINE Bus STARTING 2024-04-27T01:10:31.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: CherryPy Checker: 2024-04-27T01:10:31.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: The Application mounted at '' has an empty config. 2024-04-27T01:10:31.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: 2024-04-27T01:10:31.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: [27/Apr/2024:01:10:30] ENGINE Serving on http://:::9283 2024-04-27T01:10:31.011 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:10:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x[32090]: [27/Apr/2024:01:10:30] ENGINE Bus STARTED 2024-04-27T01:10:31.340 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:31.018+0000 7f9030f7b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-27T01:10:31.553 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:31.569 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:31.571 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:31.574 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:31.576 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:31.588 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:31.598 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:31.615 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:31.704 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:10:31] ENGINE Bus STARTING 2024-04-27T01:10:31.704 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: CherryPy Checker: 2024-04-27T01:10:31.704 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: The Application mounted at '' has an empty config. 2024-04-27T01:10:31.704 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:10:31.704 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:10:31] ENGINE Serving on http://:::9283 2024-04-27T01:10:31.704 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:10:31] ENGINE Bus STARTED 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: Standby manager daemon x restarted 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: Standby manager daemon x started 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: Active manager daemon y restarted 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: Activating manager daemon y 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: mgrmap e18: y(active, starting, since 0.0236115s), standbys: x 2024-04-27T01:10:31.705 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:10:31.706 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: Manager daemon y is now available 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: Standby manager daemon x restarted 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: Standby manager daemon x started 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-27T01:10:31.707 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:10:31.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: Active manager daemon y restarted 2024-04-27T01:10:31.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: Activating manager daemon y 2024-04-27T01:10:31.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T01:10:31.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: mgrmap e18: y(active, starting, since 0.0236115s), standbys: x 2024-04-27T01:10:31.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:10:31.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:10:31.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:10:31.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-27T01:10:31.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-27T01:10:31.708 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: Manager daemon y is now available 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:31.709 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:31.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: Standby manager daemon x restarted 2024-04-27T01:10:31.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: Standby manager daemon x started 2024-04-27T01:10:31.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-27T01:10:31.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-27T01:10:31.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-27T01:10:31.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.? v1:172.21.15.181:0/2147269782' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-27T01:10:31.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: Active manager daemon y restarted 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: Activating manager daemon y 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: osdmap e73: 8 total, 8 up, 8 in 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: mgrmap e18: y(active, starting, since 0.0236115s), standbys: x 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-27T01:10:31.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-27T01:10:31.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-27T01:10:31.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-27T01:10:31.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-27T01:10:31.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-27T01:10:31.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-27T01:10:31.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: Manager daemon y is now available 2024-04-27T01:10:31.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:31.764 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:10:32.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:10:32.612 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:32 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:32.612 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:32 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:10:32.612 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:32 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:10:32.612 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:32 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-27T01:10:32.612 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:32 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-27T01:10:32.612 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:32 smithi181 ceph-mon[30237]: mgrmap e19: y(active, since 1.01545s), standbys: x 2024-04-27T01:10:32.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:32.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:10:32.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:10:32.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-27T01:10:32.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-27T01:10:32.773 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[37680]: mgrmap e19: y(active, since 1.01545s), standbys: x 2024-04-27T01:10:32.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:32.773 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:10:32.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-27T01:10:32.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-27T01:10:32.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-27T01:10:32.774 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:32 smithi162 ceph-mon[26714]: mgrmap e19: y(active, since 1.01545s), standbys: x 2024-04-27T01:10:33.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:33 smithi162 ceph-mon[37680]: [27/Apr/2024:01:10:32] ENGINE Bus STARTING 2024-04-27T01:10:33.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:33 smithi162 ceph-mon[37680]: [27/Apr/2024:01:10:32] ENGINE Serving on https://172.21.15.162:7150 2024-04-27T01:10:33.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:33 smithi162 ceph-mon[37680]: [27/Apr/2024:01:10:32] ENGINE Serving on http://172.21.15.162:8765 2024-04-27T01:10:33.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:33 smithi162 ceph-mon[37680]: [27/Apr/2024:01:10:32] ENGINE Bus STARTED 2024-04-27T01:10:33.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:33 smithi162 ceph-mon[26714]: [27/Apr/2024:01:10:32] ENGINE Bus STARTING 2024-04-27T01:10:33.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:33 smithi162 ceph-mon[26714]: [27/Apr/2024:01:10:32] ENGINE Serving on https://172.21.15.162:7150 2024-04-27T01:10:33.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:33 smithi162 ceph-mon[26714]: [27/Apr/2024:01:10:32] ENGINE Serving on http://172.21.15.162:8765 2024-04-27T01:10:33.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:33 smithi162 ceph-mon[26714]: [27/Apr/2024:01:10:32] ENGINE Bus STARTED 2024-04-27T01:10:33.604 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:33 smithi181 ceph-mon[30237]: [27/Apr/2024:01:10:32] ENGINE Bus STARTING 2024-04-27T01:10:33.604 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:33 smithi181 ceph-mon[30237]: [27/Apr/2024:01:10:32] ENGINE Serving on https://172.21.15.162:7150 2024-04-27T01:10:33.604 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:33 smithi181 ceph-mon[30237]: [27/Apr/2024:01:10:32] ENGINE Serving on http://172.21.15.162:8765 2024-04-27T01:10:33.604 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:33 smithi181 ceph-mon[30237]: [27/Apr/2024:01:10:32] ENGINE Bus STARTED 2024-04-27T01:10:34.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:34 smithi181 ceph-mon[30237]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:10:34.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:34 smithi181 ceph-mon[30237]: mgrmap e20: y(active, since 2s), standbys: x 2024-04-27T01:10:34.667 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:34 smithi162 ceph-mon[26714]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:10:34.667 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:34 smithi162 ceph-mon[26714]: mgrmap e20: y(active, since 2s), standbys: x 2024-04-27T01:10:34.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:34 smithi162 ceph-mon[37680]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:10:34.667 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:34 smithi162 ceph-mon[37680]: mgrmap e20: y(active, since 2s), standbys: x 2024-04-27T01:10:35.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:35 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:35.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:35 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:35.924 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:35 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:35.924 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:35 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:36.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:35 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:36.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:35 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:36.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:10:36.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: mgrmap e21: y(active, since 4s), standbys: x 2024-04-27T01:10:36.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:36.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:36.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi181 to 3760M 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:10:36.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: mgrmap e21: y(active, since 4s), standbys: x 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi181 to 3760M 2024-04-27T01:10:36.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:37.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:10:37.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: mgrmap e21: y(active, since 4s), standbys: x 2024-04-27T01:10:37.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi181 to 3760M 2024-04-27T01:10:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:38.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.495+0000 7f19b10a7700 1 -- 172.21.15.162:0/3638190365 >> v1:172.21.15.162:6789/0 conn(0x7f19ac10b010 legacy=0x7f19ac10d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:38.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.495+0000 7f19b10a7700 1 -- 172.21.15.162:0/3638190365 shutdown_connections 2024-04-27T01:10:38.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.495+0000 7f19b10a7700 1 -- 172.21.15.162:0/3638190365 >> 172.21.15.162:0/3638190365 conn(0x7f19ac0fca10 msgr2=0x7f19ac0fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:38.486 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.495+0000 7f19b10a7700 1 -- 172.21.15.162:0/3638190365 shutdown_connections 2024-04-27T01:10:38.487 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.495+0000 7f19b10a7700 1 -- 172.21.15.162:0/3638190365 wait complete. 2024-04-27T01:10:38.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f19b10a7700 1 Processor -- start 2024-04-27T01:10:38.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f19b10a7700 1 -- start start 2024-04-27T01:10:38.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f19b10a7700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f19ac1a6fd0 con 0x7f19ac103200 2024-04-27T01:10:38.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f19b10a7700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f19ac1a81f0 con 0x7f19ac10b010 2024-04-27T01:10:38.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f19b10a7700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f19ac1a9430 con 0x7f19ac10eb20 2024-04-27T01:10:38.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f19aad9d700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f19ac103200 0x7f19ac10a730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:51898/0 (socket says 172.21.15.162:51898) 2024-04-27T01:10:38.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f19aad9d700 1 -- 172.21.15.162:0/2038204715 learned_addr learned my addr 172.21.15.162:0/2038204715 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:38.492 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f19ab59e700 1 --1- 172.21.15.162:0/2038204715 >> v1:172.21.15.162:6790/0 conn(0x7f19ac10eb20 0x7f19ac1a5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:59516/0 (socket says 172.21.15.162:59516) 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299469175 0 0) 0x7f19ac1a9430 con 0x7f19ac10eb20 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19940035b0 con 0x7f19ac10eb20 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892268141 0 0) 0x7f19ac1a81f0 con 0x7f19ac10b010 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.496+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19ac1a9430 con 0x7f19ac10b010 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 9705212 0 0) 0x7f19940035b0 con 0x7f19ac10eb20 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19ac1a81f0 con 0x7f19ac10eb20 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f199c0012d0 con 0x7f19ac10eb20 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 304559252 0 0) 0x7f19ac1a9430 con 0x7f19ac10b010 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f19940035b0 con 0x7f19ac10b010 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f19a0004460 con 0x7f19ac10b010 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3479257462 0 0) 0x7f19ac1a6fd0 con 0x7f19ac103200 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19ac1a9430 con 0x7f19ac103200 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 717685473 0 0) 0x7f19940035b0 con 0x7f19ac10b010 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 >> v1:172.21.15.162:6790/0 conn(0x7f19ac10eb20 legacy=0x7f19ac1a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 >> v1:172.21.15.162:6789/0 conn(0x7f19ac103200 legacy=0x7f19ac10a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ac1aa670 con 0x7f19ac10b010 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f19ac1a8420 con 0x7f19ac10b010 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f19ac1a88e0 con 0x7f19ac10b010 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f19a0001b70 con 0x7f19ac10b010 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.497+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f19a0004dc0 con 0x7f19ac10b010 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.498+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.181:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f1998000fb0 con 0x7f19ac10b010 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.499+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f19a0003860 con 0x7f19ac10b010 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.499+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7f19a0088570 con 0x7f19ac10b010 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.499+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.162:6801/2041814805 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f19a0001080 con 0x7f19940792e0 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.499+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (unknown 3489808268 0 0) 0x7f19a0088990 con 0x7f19ac10b010 2024-04-27T01:10:38.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.500+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== osd.0 v1:172.21.15.162:6801/2041814805 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f19a0001080 con 0x7f19940792e0 2024-04-27T01:10:38.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.517+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 --> v1:172.21.15.162:6801/2041814805 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1998002d10 con 0x7f19940792e0 2024-04-27T01:10:38.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.518+0000 7f1993fff700 1 -- 172.21.15.162:0/2038204715 <== osd.0 v1:172.21.15.162:6801/2041814805 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 2546019811) 0x7f1998002d10 con 0x7f19940792e0 2024-04-27T01:10:38.509 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.518+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 >> v1:172.21.15.162:6801/2041814805 conn(0x7f19940792e0 legacy=0x7f199407b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:38.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.518+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 >> v1:172.21.15.162:6800/624703537 conn(0x7f199406ec00 legacy=0x7f19940710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:38.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.518+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 >> v1:172.21.15.181:6789/0 conn(0x7f19ac10b010 legacy=0x7f19ac1a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:38.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.518+0000 7f19ab59e700 1 -- 172.21.15.162:0/2038204715 reap_dead start 2024-04-27T01:10:38.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.518+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 shutdown_connections 2024-04-27T01:10:38.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.518+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 >> 172.21.15.162:0/2038204715 conn(0x7f19ac0fca10 msgr2=0x7f19ac0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:38.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.519+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 shutdown_connections 2024-04-27T01:10:38.510 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:38.519+0000 7f19b10a7700 1 -- 172.21.15.162:0/2038204715 wait complete. 2024-04-27T01:10:38.725 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:38 smithi162 ceph-mon[37680]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-27T01:10:38.725 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:38 smithi162 ceph-mon[26714]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-27T01:10:39.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.018+0000 7f7ad7d70700 1 -- 172.21.15.162:0/3780812947 >> v1:172.21.15.162:6790/0 conn(0x7f7ad01011b0 legacy=0x7f7ad0101590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:39.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.018+0000 7f7ad7d70700 1 -- 172.21.15.162:0/3780812947 shutdown_connections 2024-04-27T01:10:39.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.018+0000 7f7ad7d70700 1 -- 172.21.15.162:0/3780812947 >> 172.21.15.162:0/3780812947 conn(0x7f7ad00fca40 msgr2=0x7f7ad00fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:39.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.018+0000 7f7ad7d70700 1 -- 172.21.15.162:0/3780812947 shutdown_connections 2024-04-27T01:10:39.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.018+0000 7f7ad7d70700 1 -- 172.21.15.162:0/3780812947 wait complete. 2024-04-27T01:10:39.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.019+0000 7f7ad7d70700 1 Processor -- start 2024-04-27T01:10:39.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.019+0000 7f7ad7d70700 1 -- start start 2024-04-27T01:10:39.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:38 smithi181 ceph-mon[30237]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-04-27T01:10:39.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.019+0000 7f7ad7d70700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7ad01a9450 con 0x7f7ad01011b0 2024-04-27T01:10:39.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.019+0000 7f7ad7d70700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7ad01aa670 con 0x7f7ad0109f50 2024-04-27T01:10:39.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.019+0000 7f7ad7d70700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7ad01ab8b0 con 0x7f7ad010daf0 2024-04-27T01:10:39.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.019+0000 7f7ad5b0c700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f7ad01011b0 0x7f7ad0106670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:51916/0 (socket says 172.21.15.162:51916) 2024-04-27T01:10:39.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.019+0000 7f7ad5b0c700 1 -- 172.21.15.162:0/4101316292 learned_addr learned my addr 172.21.15.162:0/4101316292 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:39.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3986330077 0 0) 0x7f7ad01ab8b0 con 0x7f7ad010daf0 2024-04-27T01:10:39.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7abc0035b0 con 0x7f7ad010daf0 2024-04-27T01:10:39.013 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3677849599 0 0) 0x7f7ad01a9450 con 0x7f7ad01011b0 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ad01ab8b0 con 0x7f7ad01011b0 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 597840265 0 0) 0x7f7ad01aa670 con 0x7f7ad0109f50 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ad01a9450 con 0x7f7ad0109f50 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3983325915 0 0) 0x7f7abc0035b0 con 0x7f7ad010daf0 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ad01aa670 con 0x7f7ad010daf0 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7ac8001590 con 0x7f7ad010daf0 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1696123486 0 0) 0x7f7ad01ab8b0 con 0x7f7ad01011b0 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7abc0035b0 con 0x7f7ad01011b0 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7acc0044f0 con 0x7f7ad01011b0 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3292166260 0 0) 0x7f7ad01a9450 con 0x7f7ad0109f50 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ad01ab8b0 con 0x7f7ad0109f50 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7ac0001830 con 0x7f7ad0109f50 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1290885271 0 0) 0x7f7ad01aa670 con 0x7f7ad010daf0 2024-04-27T01:10:39.014 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 >> v1:172.21.15.181:6789/0 conn(0x7f7ad0109f50 legacy=0x7f7ad01a4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 >> v1:172.21.15.162:6789/0 conn(0x7f7ad01011b0 legacy=0x7f7ad0106670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad01acaf0 con 0x7f7ad010daf0 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad01a9680 con 0x7f7ad010daf0 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7ad01a9b40 con 0x7f7ad010daf0 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f7ac8003e90 con 0x7f7ad010daf0 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.020+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7ac8004cf0 con 0x7f7ad010daf0 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.021+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.162:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f7ab4000f80 con 0x7f7ad010daf0 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.022+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f7ac8003a00 con 0x7f7ad010daf0 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.022+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7f7ac8088500 con 0x7f7ad010daf0 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.022+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.162:6813/2875926639 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7abc07d390 con 0x7f7abc079680 2024-04-27T01:10:39.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.022+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (unknown 3489808268 0 0) 0x7f7ac8088910 con 0x7f7ad010daf0 2024-04-27T01:10:39.016 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.023+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== osd.3 v1:172.21.15.162:6813/2875926639 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7abc07d390 con 0x7f7abc079680 2024-04-27T01:10:39.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.039+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 --> v1:172.21.15.162:6813/2875926639 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7ab4002ce0 con 0x7f7abc079680 2024-04-27T01:10:39.030 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.040+0000 7f7ac6ffd700 1 -- 172.21.15.162:0/4101316292 <== osd.3 v1:172.21.15.162:6813/2875926639 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 1221308824) 0x7f7ab4002ce0 con 0x7f7abc079680 2024-04-27T01:10:39.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.040+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 >> v1:172.21.15.162:6813/2875926639 conn(0x7f7abc079680 legacy=0x7f7abc07bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:39.031 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.040+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 >> v1:172.21.15.162:6800/624703537 conn(0x7f7abc06ef20 legacy=0x7f7abc0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:39.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.040+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 >> v1:172.21.15.162:6790/0 conn(0x7f7ad010daf0 legacy=0x7f7ad01a7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:39.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.040+0000 7f7ad630d700 1 -- 172.21.15.162:0/4101316292 reap_dead start 2024-04-27T01:10:39.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.040+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 shutdown_connections 2024-04-27T01:10:39.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.040+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 >> 172.21.15.162:0/4101316292 conn(0x7f7ad00fca40 msgr2=0x7f7ad0110760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:39.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.040+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 shutdown_connections 2024-04-27T01:10:39.032 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:39.040+0000 7f7ad7d70700 1 -- 172.21.15.162:0/4101316292 wait complete. 2024-04-27T01:10:39.950 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:10:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:10:40.498 INFO:teuthology.orchestra.run.smithi162.stdout:34359738416 2024-04-27T01:10:40.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-27T01:10:40.672 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:40 smithi162 ceph-mon[26714]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-27T01:10:40.938 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:40 smithi162 ceph-mon[37680]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-27T01:10:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:40 smithi181 ceph-mon[30237]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-27T01:10:41.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.857+0000 7f67547e0700 1 -- 172.21.15.162:0/3736423532 >> v1:172.21.15.162:6790/0 conn(0x7f674c10aee0 legacy=0x7f674c10d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.857+0000 7f67547e0700 1 -- 172.21.15.162:0/3736423532 shutdown_connections 2024-04-27T01:10:41.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.857+0000 7f67547e0700 1 -- 172.21.15.162:0/3736423532 >> 172.21.15.162:0/3736423532 conn(0x7f674c0fdb20 msgr2=0x7f674c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.857+0000 7f67547e0700 1 -- 172.21.15.162:0/3736423532 shutdown_connections 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.857+0000 7f67547e0700 1 -- 172.21.15.162:0/3736423532 wait complete. 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.857+0000 7f67547e0700 1 Processor -- start 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67547e0700 1 -- start start 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67547e0700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f674c1a7040 con 0x7f674c10ea80 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67547e0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f674c1a8260 con 0x7f674c10aee0 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67547e0700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f674c1a94a0 con 0x7f674c102250 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f675257c700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f674c102250 0x7f674c1074e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:59570/0 (socket says 172.21.15.162:59570) 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f675257c700 1 -- 172.21.15.162:0/3097327485 learned_addr learned my addr 172.21.15.162:0/3097327485 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1850660082 0 0) 0x7f674c1a94a0 con 0x7f674c102250 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67380035b0 con 0x7f674c102250 2024-04-27T01:10:41.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2338995934 0 0) 0x7f674c1a7040 con 0x7f674c10ea80 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f674c1a94a0 con 0x7f674c10ea80 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1568192009 0 0) 0x7f674c1a8260 con 0x7f674c10aee0 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f674c1a7040 con 0x7f674c10aee0 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3334483292 0 0) 0x7f67380035b0 con 0x7f674c102250 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f674c1a8260 con 0x7f674c102250 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6748001930 con 0x7f674c102250 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1612096541 0 0) 0x7f674c1a94a0 con 0x7f674c10ea80 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67380035b0 con 0x7f674c10ea80 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6744001b70 con 0x7f674c10ea80 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.858+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2821451326 0 0) 0x7f674c1a7040 con 0x7f674c10aee0 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f674c1a94a0 con 0x7f674c10aee0 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f673c0044e0 con 0x7f674c10aee0 2024-04-27T01:10:41.867 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1973869874 0 0) 0x7f674c1a8260 con 0x7f674c102250 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 >> v1:172.21.15.181:6789/0 conn(0x7f674c10aee0 legacy=0x7f674c1a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 >> v1:172.21.15.162:6789/0 conn(0x7f674c10ea80 legacy=0x7f674c1a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c1aa6e0 con 0x7f674c102250 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f674c1a7270 con 0x7f674c102250 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f674c1a7730 con 0x7f674c102250 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f6748003b30 con 0x7f674c102250 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6748005cc0 con 0x7f674c102250 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.859+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.162:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6730000f80 con 0x7f674c102250 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.860+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f674801c700 con 0x7f674c102250 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.861+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7f67480896e0 con 0x7f674c102250 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.861+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.162:6805/680367434 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f673c001080 con 0x7f6738079670 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.861+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (unknown 3489808268 0 0) 0x7f6748089af0 con 0x7f674c102250 2024-04-27T01:10:41.868 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.862+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== osd.1 v1:172.21.15.162:6805/680367434 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f673c001080 con 0x7f6738079670 2024-04-27T01:10:41.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.878+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 --> v1:172.21.15.162:6805/680367434 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6730002ce0 con 0x7f6738079670 2024-04-27T01:10:41.869 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.879+0000 7f67437fe700 1 -- 172.21.15.162:0/3097327485 <== osd.1 v1:172.21.15.162:6805/680367434 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 4200525082) 0x7f6730002ce0 con 0x7f6738079670 2024-04-27T01:10:41.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.879+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 >> v1:172.21.15.162:6805/680367434 conn(0x7f6738079670 legacy=0x7f673807bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.870 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.879+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 >> v1:172.21.15.162:6800/624703537 conn(0x7f673806ef90 legacy=0x7f6738071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.879+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 >> v1:172.21.15.162:6790/0 conn(0x7f674c102250 legacy=0x7f674c1074e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.879+0000 7f6752d7d700 1 -- 172.21.15.162:0/3097327485 reap_dead start 2024-04-27T01:10:41.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.879+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 shutdown_connections 2024-04-27T01:10:41.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.879+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 >> 172.21.15.162:0/3097327485 conn(0x7f674c0fdb20 msgr2=0x7f674c111f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:41.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.879+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 shutdown_connections 2024-04-27T01:10:41.871 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.880+0000 7f67547e0700 1 -- 172.21.15.162:0/3097327485 wait complete. 2024-04-27T01:10:41.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.765+0000 7fe2ad9de700 1 -- 172.21.15.162:0/203448700 >> v1:172.21.15.162:6789/0 conn(0x7fe2a810af10 legacy=0x7fe2a810d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.765+0000 7fe2ad9de700 1 -- 172.21.15.162:0/203448700 shutdown_connections 2024-04-27T01:10:41.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.765+0000 7fe2ad9de700 1 -- 172.21.15.162:0/203448700 >> 172.21.15.162:0/203448700 conn(0x7fe2a80fdb50 msgr2=0x7fe2a80fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:41.982 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.765+0000 7fe2ad9de700 1 -- 172.21.15.162:0/203448700 shutdown_connections 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.765+0000 7fe2ad9de700 1 -- 172.21.15.162:0/203448700 wait complete. 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.766+0000 7fe2ad9de700 1 Processor -- start 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.766+0000 7fe2ad9de700 1 -- start start 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.766+0000 7fe2ad9de700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe2a8107950 con 0x7fe2a810eab0 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.766+0000 7fe2ad9de700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe2a8107b60 con 0x7fe2a810af10 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.766+0000 7fe2ad9de700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe2a81a96d0 con 0x7fe2a8102280 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.766+0000 7fe2a77fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe2a810eab0 0x7fe2a81a5f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:51940/0 (socket says 172.21.15.162:51940) 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.766+0000 7fe2a6ffd700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fe2a8102280 0x7fe2a81047a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:59552/0 (socket says 172.21.15.162:59552) 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.766+0000 7fe2a77fe700 1 -- 172.21.15.162:0/3679096391 learned_addr learned my addr 172.21.15.162:0/3679096391 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292690356 0 0) 0x7fe2a81a96d0 con 0x7fe2a8102280 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2900035b0 con 0x7fe2a8102280 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 853248980 0 0) 0x7fe2a8107950 con 0x7fe2a810eab0 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2a81a96d0 con 0x7fe2a810eab0 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388878088 0 0) 0x7fe2a8107b60 con 0x7fe2a810af10 2024-04-27T01:10:41.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2a8107950 con 0x7fe2a810af10 2024-04-27T01:10:41.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 44416057 0 0) 0x7fe2a81a96d0 con 0x7fe2a810eab0 2024-04-27T01:10:41.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2a8107b60 con 0x7fe2a810eab0 2024-04-27T01:10:41.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fe2a0001b40 con 0x7fe2a810eab0 2024-04-27T01:10:41.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2297348155 0 0) 0x7fe2900035b0 con 0x7fe2a8102280 2024-04-27T01:10:41.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2a81a96d0 con 0x7fe2a8102280 2024-04-27T01:10:41.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fe2980018b0 con 0x7fe2a8102280 2024-04-27T01:10:41.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1139879571 0 0) 0x7fe2a8107b60 con 0x7fe2a810eab0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 >> v1:172.21.15.162:6790/0 conn(0x7fe2a8102280 legacy=0x7fe2a81047a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 >> v1:172.21.15.181:6789/0 conn(0x7fe2a810af10 legacy=0x7fe2a8104eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a81aa8f0 con 0x7fe2a810eab0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a81a9900 con 0x7fe2a810eab0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.767+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe2a81a9dc0 con 0x7fe2a810eab0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.768+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fe2a0003eb0 con 0x7fe2a810eab0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.768+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fe2a0005f10 con 0x7fe2a810eab0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.768+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.162:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe294000f80 con 0x7fe2a810eab0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.769+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7fe2a0003a20 con 0x7fe2a810eab0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.770+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7fe2a0089950 con 0x7fe2a810eab0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.770+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.181:6804/3172409639 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe29007d3c0 con 0x7fe2900796b0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.770+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (unknown 3489808268 0 0) 0x7fe2a0089dc0 con 0x7fe2a810eab0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.771+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== osd.5 v1:172.21.15.181:6804/3172409639 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe29007d3c0 con 0x7fe2900796b0 2024-04-27T01:10:41.985 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.806+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 --> v1:172.21.15.181:6804/3172409639 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe294002ce0 con 0x7fe2900796b0 2024-04-27T01:10:41.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.806+0000 7fe2ac9dc700 1 -- 172.21.15.162:0/3679096391 <== osd.5 v1:172.21.15.181:6804/3172409639 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 1238763653) 0x7fe294002ce0 con 0x7fe2900796b0 2024-04-27T01:10:41.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.807+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 >> v1:172.21.15.181:6804/3172409639 conn(0x7fe2900796b0 legacy=0x7fe29007bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.807+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 >> v1:172.21.15.162:6800/624703537 conn(0x7fe29006eed0 legacy=0x7fe290071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.807+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 >> v1:172.21.15.162:6789/0 conn(0x7fe2a810eab0 legacy=0x7fe2a81a5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:41.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.807+0000 7fe2a77fe700 1 -- 172.21.15.162:0/3679096391 reap_dead start 2024-04-27T01:10:41.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.807+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 shutdown_connections 2024-04-27T01:10:41.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.807+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 >> 172.21.15.162:0/3679096391 conn(0x7fe2a80fdb50 msgr2=0x7fe2a8111700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:41.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.807+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 shutdown_connections 2024-04-27T01:10:41.986 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:41.807+0000 7fe2ad9de700 1 -- 172.21.15.162:0/3679096391 wait complete. 2024-04-27T01:10:42.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.382+0000 7fe18be3c700 1 -- 172.21.15.162:0/4261377159 >> v1:172.21.15.162:6789/0 conn(0x7fe184101b70 legacy=0x7fe184101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:42.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.382+0000 7fe18be3c700 1 -- 172.21.15.162:0/4261377159 shutdown_connections 2024-04-27T01:10:42.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.382+0000 7fe18be3c700 1 -- 172.21.15.162:0/4261377159 >> 172.21.15.162:0/4261377159 conn(0x7fe184076220 msgr2=0x7fe184076630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:42.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.382+0000 7fe18be3c700 1 -- 172.21.15.162:0/4261377159 shutdown_connections 2024-04-27T01:10:42.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.382+0000 7fe18be3c700 1 -- 172.21.15.162:0/4261377159 wait complete. 2024-04-27T01:10:42.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe18be3c700 1 Processor -- start 2024-04-27T01:10:42.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe18be3c700 1 -- start start 2024-04-27T01:10:42.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe18be3c700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe1841a9140 con 0x7fe184109e50 2024-04-27T01:10:42.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe18be3c700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe1841aa320 con 0x7fe18410d9f0 2024-04-27T01:10:42.376 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe18be3c700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fe1841ab540 con 0x7fe184101b70 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe189bd8700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fe184101b70 0x7fe1841061f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:59582/0 (socket says 172.21.15.162:59582) 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe189bd8700 1 -- 172.21.15.162:0/4051594071 learned_addr learned my addr 172.21.15.162:0/4051594071 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2909023763 0 0) 0x7fe1841ab540 con 0x7fe184101b70 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1700035b0 con 0x7fe184101b70 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2412817231 0 0) 0x7fe1841a9140 con 0x7fe184109e50 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1841ab540 con 0x7fe184109e50 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3136756229 0 0) 0x7fe1841aa320 con 0x7fe18410d9f0 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.383+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe1841a9140 con 0x7fe18410d9f0 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4201262824 0 0) 0x7fe1700035b0 con 0x7fe184101b70 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1841aa320 con 0x7fe184101b70 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 704037525 0 0) 0x7fe1841ab540 con 0x7fe184109e50 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1700035b0 con 0x7fe184109e50 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fe180004580 con 0x7fe184101b70 2024-04-27T01:10:42.377 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fe1740017b0 con 0x7fe184109e50 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2808732236 0 0) 0x7fe1841a9140 con 0x7fe18410d9f0 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe1841ab540 con 0x7fe18410d9f0 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fe17c0013f0 con 0x7fe18410d9f0 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2501868208 0 0) 0x7fe1841aa320 con 0x7fe184101b70 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 >> v1:172.21.15.181:6789/0 conn(0x7fe18410d9f0 legacy=0x7fe1841a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 >> v1:172.21.15.162:6789/0 conn(0x7fe184109e50 legacy=0x7fe1841a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1841ac7a0 con 0x7fe184101b70 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe1841a9370 con 0x7fe184101b70 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe1841a9830 con 0x7fe184101b70 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.384+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fe180003c10 con 0x7fe184101b70 2024-04-27T01:10:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.385+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fe180005140 con 0x7fe184101b70 2024-04-27T01:10:42.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.385+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.162:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe1840698f0 con 0x7fe184101b70 2024-04-27T01:10:42.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.385+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7fe1800053c0 con 0x7fe184101b70 2024-04-27T01:10:42.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.386+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7fe1800896e0 con 0x7fe184101b70 2024-04-27T01:10:42.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.386+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.181:6808/30247032 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe180001d10 con 0x7fe170079760 2024-04-27T01:10:42.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.386+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (unknown 3489808268 0 0) 0x7fe180089b50 con 0x7fe184101b70 2024-04-27T01:10:42.379 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.388+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== osd.6 v1:172.21.15.181:6808/30247032 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe180001d10 con 0x7fe170079760 2024-04-27T01:10:42.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.404+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 --> v1:172.21.15.181:6808/30247032 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe1841093b0 con 0x7fe170079760 2024-04-27T01:10:42.395 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.404+0000 7fe17affd700 1 -- 172.21.15.162:0/4051594071 <== osd.6 v1:172.21.15.181:6808/30247032 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 992237111) 0x7fe1841093b0 con 0x7fe170079760 2024-04-27T01:10:42.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.405+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 >> v1:172.21.15.181:6808/30247032 conn(0x7fe170079760 legacy=0x7fe17007bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:42.396 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.405+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 >> v1:172.21.15.162:6800/624703537 conn(0x7fe17006ef70 legacy=0x7fe170071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:42.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.405+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 >> v1:172.21.15.162:6790/0 conn(0x7fe184101b70 legacy=0x7fe1841061f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:42.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.405+0000 7fe18a3d9700 1 -- 172.21.15.162:0/4051594071 reap_dead start 2024-04-27T01:10:42.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.405+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 shutdown_connections 2024-04-27T01:10:42.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.405+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 >> 172.21.15.162:0/4051594071 conn(0x7fe184076220 msgr2=0x7fe184110ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:42.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.405+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 shutdown_connections 2024-04-27T01:10:42.397 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:42.405+0000 7fe18be3c700 1 -- 172.21.15.162:0/4051594071 wait complete. 2024-04-27T01:10:42.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:10:42.789 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:42 smithi162 ceph-mon[37680]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-27T01:10:42.789 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:42 smithi162 ceph-mon[26714]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-27T01:10:42.790 INFO:teuthology.orchestra.run.smithi162.stdout:167503724567 2024-04-27T01:10:42.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-27T01:10:43.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:42 smithi181 ceph-mon[30237]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-27T01:10:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:10:44.026 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:10:44.027 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:10:44.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.143+0000 7fce93f12700 1 -- 172.21.15.162:0/1511154566 >> v1:172.21.15.162:6789/0 conn(0x7fce8c109df0 legacy=0x7fce8c10c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:44.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.143+0000 7fce93f12700 1 -- 172.21.15.162:0/1511154566 shutdown_connections 2024-04-27T01:10:44.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.143+0000 7fce93f12700 1 -- 172.21.15.162:0/1511154566 >> 172.21.15.162:0/1511154566 conn(0x7fce8c0fc9f0 msgr2=0x7fce8c0fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:44.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.143+0000 7fce93f12700 1 -- 172.21.15.162:0/1511154566 shutdown_connections 2024-04-27T01:10:44.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.143+0000 7fce93f12700 1 -- 172.21.15.162:0/1511154566 wait complete. 2024-04-27T01:10:44.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce93f12700 1 Processor -- start 2024-04-27T01:10:44.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce93f12700 1 -- start start 2024-04-27T01:10:44.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce93f12700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fce8c1a5eb0 con 0x7fce8c109df0 2024-04-27T01:10:44.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce93f12700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fce8c1a70d0 con 0x7fce8c101160 2024-04-27T01:10:44.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce93f12700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fce8c1a8310 con 0x7fce8c10d990 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce914ad700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fce8c109df0 0x7fce8c1a0e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52008/0 (socket says 172.21.15.162:52008) 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce914ad700 1 -- 172.21.15.162:0/1993890700 learned_addr learned my addr 172.21.15.162:0/1993890700 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3583984768 0 0) 0x7fce8c1a5eb0 con 0x7fce8c109df0 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce780035b0 con 0x7fce8c109df0 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793610765 0 0) 0x7fce8c1a8310 con 0x7fce8c10d990 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.144+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce8c1a5eb0 con 0x7fce8c10d990 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2513189611 0 0) 0x7fce8c1a70d0 con 0x7fce8c101160 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce8c1a8310 con 0x7fce8c101160 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1637638228 0 0) 0x7fce780035b0 con 0x7fce8c109df0 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce8c1a70d0 con 0x7fce8c109df0 2024-04-27T01:10:44.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fce7c004490 con 0x7fce8c109df0 2024-04-27T01:10:44.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2036214017 0 0) 0x7fce8c1a5eb0 con 0x7fce8c10d990 2024-04-27T01:10:44.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce780035b0 con 0x7fce8c10d990 2024-04-27T01:10:44.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fce84001b20 con 0x7fce8c10d990 2024-04-27T01:10:44.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1504278375 0 0) 0x7fce8c1a8310 con 0x7fce8c101160 2024-04-27T01:10:44.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce8c1a5eb0 con 0x7fce8c101160 2024-04-27T01:10:44.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fce880018b0 con 0x7fce8c101160 2024-04-27T01:10:44.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 689469876 0 0) 0x7fce8c1a70d0 con 0x7fce8c109df0 2024-04-27T01:10:44.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 >> v1:172.21.15.162:6790/0 conn(0x7fce8c10d990 legacy=0x7fce8c1a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:44.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 >> v1:172.21.15.181:6789/0 conn(0x7fce8c101160 legacy=0x7fce8c106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:44.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce8c1a9550 con 0x7fce8c109df0 2024-04-27T01:10:44.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fce8c1a60e0 con 0x7fce8c109df0 2024-04-27T01:10:44.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fce8c1a65a0 con 0x7fce8c109df0 2024-04-27T01:10:44.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fce7c001190 con 0x7fce8c109df0 2024-04-27T01:10:44.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.145+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fce7c004c80 con 0x7fce8c109df0 2024-04-27T01:10:44.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.146+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.162:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fce70000f80 con 0x7fce8c109df0 2024-04-27T01:10:44.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.146+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7fce7c001a60 con 0x7fce8c109df0 2024-04-27T01:10:44.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.147+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7fce7c0893f0 con 0x7fce8c109df0 2024-04-27T01:10:44.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.147+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.181:6800/1869417285 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce7807f6b0 con 0x7fce7807b9a0 2024-04-27T01:10:44.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.147+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (unknown 3489808268 0 0) 0x7fce7c089860 con 0x7fce8c109df0 2024-04-27T01:10:44.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.148+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== osd.4 v1:172.21.15.181:6800/1869417285 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fce7807f6b0 con 0x7fce7807b9a0 2024-04-27T01:10:44.154 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.164+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 --> v1:172.21.15.181:6800/1869417285 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fce70002ce0 con 0x7fce7807b9a0 2024-04-27T01:10:44.155 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.165+0000 7fce82ffd700 1 -- 172.21.15.162:0/1993890700 <== osd.4 v1:172.21.15.181:6800/1869417285 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 2875114804) 0x7fce70002ce0 con 0x7fce7807b9a0 2024-04-27T01:10:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.165+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 >> v1:172.21.15.181:6800/1869417285 conn(0x7fce7807b9a0 legacy=0x7fce7807de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.165+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 >> v1:172.21.15.162:6800/624703537 conn(0x7fce78071140 legacy=0x7fce78073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.165+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 >> v1:172.21.15.162:6789/0 conn(0x7fce8c109df0 legacy=0x7fce8c1a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.165+0000 7fce924af700 1 -- 172.21.15.162:0/1993890700 reap_dead start 2024-04-27T01:10:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.165+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 shutdown_connections 2024-04-27T01:10:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.165+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 >> 172.21.15.162:0/1993890700 conn(0x7fce8c0fc9f0 msgr2=0x7fce8c106cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.165+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 shutdown_connections 2024-04-27T01:10:44.156 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:44.165+0000 7fce93f12700 1 -- 172.21.15.162:0/1993890700 wait complete. 2024-04-27T01:10:44.681 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:44 smithi162 ceph-mon[37680]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-27T01:10:44.682 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:44 smithi162 ceph-mon[26714]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-27T01:10:44.714 INFO:teuthology.orchestra.run.smithi162.stdout:201863462931 2024-04-27T01:10:44.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-27T01:10:45.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:44 smithi181 ceph-mon[30237]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-27T01:10:45.465 INFO:teuthology.orchestra.run.smithi162.stdout:133143986205 2024-04-27T01:10:45.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-27T01:10:46.599 INFO:teuthology.orchestra.run.smithi162.stdout:111669149730 2024-04-27T01:10:46.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-27T01:10:46.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:46 smithi162 ceph-mon[26714]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T01:10:46.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:46 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:46.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:46 smithi162 ceph-mon[37680]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T01:10:46.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:46 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:47.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:46 smithi181 ceph-mon[30237]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T01:10:47.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:46 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:10:47.771 INFO:teuthology.orchestra.run.smithi162.stdout:55834574892 2024-04-27T01:10:47.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-27T01:10:48.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.260+0000 7f5be49ac700 1 -- 172.21.15.162:0/2476955617 >> v1:172.21.15.162:6790/0 conn(0x7f5be0101160 legacy=0x7f5be0101540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.260+0000 7f5be49ac700 1 -- 172.21.15.162:0/2476955617 shutdown_connections 2024-04-27T01:10:48.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.260+0000 7f5be49ac700 1 -- 172.21.15.162:0/2476955617 >> 172.21.15.162:0/2476955617 conn(0x7f5be00fca10 msgr2=0x7f5be00fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:48.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.260+0000 7f5be49ac700 1 -- 172.21.15.162:0/2476955617 shutdown_connections 2024-04-27T01:10:48.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.260+0000 7f5be49ac700 1 -- 172.21.15.162:0/2476955617 wait complete. 2024-04-27T01:10:48.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5be49ac700 1 Processor -- start 2024-04-27T01:10:48.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5be49ac700 1 -- start start 2024-04-27T01:10:48.252 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5be49ac700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5be01a9380 con 0x7f5be0109f00 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5be49ac700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5be01aa5a0 con 0x7f5be0101160 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5be49ac700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f5be01ab7e0 con 0x7f5be010daa0 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5bded9d700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f5be010daa0 0x7f5be01a7a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:51956/0 (socket says 172.21.15.162:51956) 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5bd7fff700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f5be0109f00 0x7f5be01a4310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39774/0 (socket says 172.21.15.162:39774) 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5bded9d700 1 -- 172.21.15.162:0/3199813135 learned_addr learned my addr 172.21.15.162:0/3199813135 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3821057172 0 0) 0x7f5be01ab7e0 con 0x7f5be010daa0 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bcc0031f0 con 0x7f5be010daa0 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3382675937 0 0) 0x7f5be01a9380 con 0x7f5be0109f00 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.261+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5be01ab7e0 con 0x7f5be0109f00 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 833871716 0 0) 0x7f5be01aa5a0 con 0x7f5be0101160 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5be01a9380 con 0x7f5be0101160 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1560836985 0 0) 0x7f5be01ab7e0 con 0x7f5be0109f00 2024-04-27T01:10:48.253 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5be01aa5a0 con 0x7f5be0109f00 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2733329299 0 0) 0x7f5bcc0031f0 con 0x7f5be010daa0 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5be01ab7e0 con 0x7f5be010daa0 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f5bc8001830 con 0x7f5be0109f00 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f5bd8001c10 con 0x7f5be010daa0 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 863145535 0 0) 0x7f5be01aa5a0 con 0x7f5be0109f00 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 >> v1:172.21.15.162:6790/0 conn(0x7f5be010daa0 legacy=0x7f5be01a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 >> v1:172.21.15.181:6789/0 conn(0x7f5be0101160 legacy=0x7f5be0106410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be01aca20 con 0x7f5be0109f00 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5be01a95b0 con 0x7f5be0109f00 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5be01a9a70 con 0x7f5be0109f00 2024-04-27T01:10:48.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f5bc8003b30 con 0x7f5be0109f00 2024-04-27T01:10:48.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.262+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f5bc8005b70 con 0x7f5be0109f00 2024-04-27T01:10:48.255 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.263+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.162:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f5bc0000f80 con 0x7f5be0109f00 2024-04-27T01:10:48.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.263+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f5bc80036a0 con 0x7f5be0109f00 2024-04-27T01:10:48.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.264+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7f5bc80894a0 con 0x7f5be0109f00 2024-04-27T01:10:48.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.264+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.162:6809/2147722624 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5bcc07d010 con 0x7f5bcc079300 2024-04-27T01:10:48.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.264+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (unknown 3489808268 0 0) 0x7f5bc8089910 con 0x7f5be0109f00 2024-04-27T01:10:48.256 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.265+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== osd.2 v1:172.21.15.162:6809/2147722624 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5bcc07d010 con 0x7f5bcc079300 2024-04-27T01:10:48.271 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.281+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 --> v1:172.21.15.162:6809/2147722624 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5bc0002ce0 con 0x7f5bcc079300 2024-04-27T01:10:48.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.282+0000 7f5bd77fe700 1 -- 172.21.15.162:0/3199813135 <== osd.2 v1:172.21.15.162:6809/2147722624 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3472381573) 0x7f5bc0002ce0 con 0x7f5bcc079300 2024-04-27T01:10:48.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.282+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 >> v1:172.21.15.162:6809/2147722624 conn(0x7f5bcc079300 legacy=0x7f5bcc07b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.282+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 >> v1:172.21.15.162:6800/624703537 conn(0x7f5bcc06ec20 legacy=0x7f5bcc0710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.282+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 >> v1:172.21.15.162:6789/0 conn(0x7f5be0109f00 legacy=0x7f5be01a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.282+0000 7f5bded9d700 1 -- 172.21.15.162:0/3199813135 reap_dead start 2024-04-27T01:10:48.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.282+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 shutdown_connections 2024-04-27T01:10:48.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.282+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 >> 172.21.15.162:0/3199813135 conn(0x7f5be00fca10 msgr2=0x7f5be006a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:48.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.282+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 shutdown_connections 2024-04-27T01:10:48.273 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.282+0000 7f5be49ac700 1 -- 172.21.15.162:0/3199813135 wait complete. 2024-04-27T01:10:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:48 smithi162 ceph-mon[37680]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T01:10:48.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:48 smithi162 ceph-mon[26714]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T01:10:48.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.503+0000 7fbd569df700 1 -- 172.21.15.162:0/1492521716 >> v1:172.21.15.162:6790/0 conn(0x7fbd5010afa0 legacy=0x7fbd5010d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.503+0000 7fbd569df700 1 -- 172.21.15.162:0/1492521716 shutdown_connections 2024-04-27T01:10:48.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.503+0000 7fbd569df700 1 -- 172.21.15.162:0/1492521716 >> 172.21.15.162:0/1492521716 conn(0x7fbd50076220 msgr2=0x7fbd50076630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:48.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.503+0000 7fbd569df700 1 -- 172.21.15.162:0/1492521716 shutdown_connections 2024-04-27T01:10:48.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.503+0000 7fbd569df700 1 -- 172.21.15.162:0/1492521716 wait complete. 2024-04-27T01:10:48.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.504+0000 7fbd569df700 1 Processor -- start 2024-04-27T01:10:48.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.504+0000 7fbd569df700 1 -- start start 2024-04-27T01:10:48.902 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.504+0000 7fbd569df700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbd501a7100 con 0x7fbd5010eba0 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.504+0000 7fbd569df700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbd501a8320 con 0x7fbd5010afa0 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.504+0000 7fbd569df700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fbd501a9560 con 0x7fbd501081a0 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.504+0000 7fbd4ffff700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fbd501081a0 0x7fbd50100f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:51964/0 (socket says 172.21.15.162:51964) 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.504+0000 7fbd4ffff700 1 -- 172.21.15.162:0/3794998848 learned_addr learned my addr 172.21.15.162:0/3794998848 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1864180936 0 0) 0x7fbd501a7100 con 0x7fbd5010eba0 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd380031f0 con 0x7fbd5010eba0 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 686249800 0 0) 0x7fbd501a9560 con 0x7fbd501081a0 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd501a7100 con 0x7fbd501081a0 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1947063351 0 0) 0x7fbd501a8320 con 0x7fbd5010afa0 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd501a9560 con 0x7fbd5010afa0 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3163873013 0 0) 0x7fbd380031f0 con 0x7fbd5010eba0 2024-04-27T01:10:48.903 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd501a8320 con 0x7fbd5010eba0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fbd48001590 con 0x7fbd5010eba0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 404374854 0 0) 0x7fbd501a7100 con 0x7fbd501081a0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd380031f0 con 0x7fbd501081a0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fbd40001870 con 0x7fbd501081a0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4064250291 0 0) 0x7fbd501a9560 con 0x7fbd5010afa0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbd501a7100 con 0x7fbd5010afa0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fbd44004500 con 0x7fbd5010afa0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3765867658 0 0) 0x7fbd501a8320 con 0x7fbd5010eba0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 >> v1:172.21.15.162:6790/0 conn(0x7fbd501081a0 legacy=0x7fbd50100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 >> v1:172.21.15.181:6789/0 conn(0x7fbd5010afa0 legacy=0x7fbd501a2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd501aa7a0 con 0x7fbd5010eba0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbd501a9790 con 0x7fbd5010eba0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbd501a9c50 con 0x7fbd5010eba0 2024-04-27T01:10:48.904 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fbd48003e90 con 0x7fbd5010eba0 2024-04-27T01:10:48.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.505+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fbd48004da0 con 0x7fbd5010eba0 2024-04-27T01:10:48.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.506+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.162:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbd3c000f80 con 0x7fbd5010eba0 2024-04-27T01:10:48.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.507+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7fbd48003a00 con 0x7fbd5010eba0 2024-04-27T01:10:48.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.507+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7fbd48088620 con 0x7fbd5010eba0 2024-04-27T01:10:48.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.507+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.181:6812/1269032985 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd3807d370 con 0x7fbd38079660 2024-04-27T01:10:48.905 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.507+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_get_version_reply(handle=1 version=73) ==== 24+0+0 (unknown 3489808268 0 0) 0x7fbd48088a30 con 0x7fbd5010eba0 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.508+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== osd.7 v1:172.21.15.181:6812/1269032985 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbd3807d370 con 0x7fbd38079660 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.524+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 --> v1:172.21.15.181:6812/1269032985 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbd3c002ce0 con 0x7fbd38079660 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.525+0000 7fbd4d7fa700 1 -- 172.21.15.162:0/3794998848 <== osd.7 v1:172.21.15.181:6812/1269032985 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 2936409065) 0x7fbd3c002ce0 con 0x7fbd38079660 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.525+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 >> v1:172.21.15.181:6812/1269032985 conn(0x7fbd38079660 legacy=0x7fbd3807bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.525+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 >> v1:172.21.15.162:6800/624703537 conn(0x7fbd3806ef80 legacy=0x7fbd38071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.525+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 >> v1:172.21.15.162:6789/0 conn(0x7fbd5010eba0 legacy=0x7fbd501a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.525+0000 7fbd54f7c700 1 -- 172.21.15.162:0/3794998848 reap_dead start 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.525+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 shutdown_connections 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.525+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 >> 172.21.15.162:0/3794998848 conn(0x7fbd50076220 msgr2=0x7fbd50112250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.525+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 shutdown_connections 2024-04-27T01:10:48.906 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:48.526+0000 7fbd569df700 1 -- 172.21.15.162:0/3794998848 wait complete. 2024-04-27T01:10:49.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:48 smithi181 ceph-mon[30237]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-27T01:10:49.520 INFO:teuthology.orchestra.run.smithi162.stdout:236223201296 2024-04-27T01:10:49.520 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-27T01:10:49.839 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:10:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:10:49.941 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:49.942 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:49.987 INFO:teuthology.orchestra.run.smithi162.stdout:77309411369 2024-04-27T01:10:49.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-27T01:10:49.996 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:49.997 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:49.997 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:50.966 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:50 smithi162 ceph-mon[37680]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:10:50.967 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:50 smithi162 ceph-mon[26714]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:10:50.977 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:50 smithi181 ceph-mon[30237]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:10:52.040 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:10:52.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:10:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:10:52.942 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:52 smithi162 ceph-mon[37680]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:10:52.942 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:52 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:52.942 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:52 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:52.942 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:52 smithi162 ceph-mon[26714]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:10:52.942 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:52 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:52.942 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:52 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:52 smithi181 ceph-mon[30237]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:10:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:52 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:52 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:53.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:10:53.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:10:54.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.112+0000 7feb6f76a700 1 -- 172.21.15.162:0/2129966023 >> v1:172.21.15.162:6790/0 conn(0x7feb68101140 legacy=0x7feb68101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.112+0000 7feb6f76a700 1 -- 172.21.15.162:0/2129966023 shutdown_connections 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.112+0000 7feb6f76a700 1 -- 172.21.15.162:0/2129966023 >> 172.21.15.162:0/2129966023 conn(0x7feb680fca10 msgr2=0x7feb680fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.112+0000 7feb6f76a700 1 -- 172.21.15.162:0/2129966023 shutdown_connections 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.112+0000 7feb6f76a700 1 -- 172.21.15.162:0/2129966023 wait complete. 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb6f76a700 1 Processor -- start 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb6f76a700 1 -- start start 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb6f76a700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feb681a7090 con 0x7feb6810d970 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb6f76a700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feb681a82b0 con 0x7feb68101140 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb6f76a700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feb681a94f0 con 0x7feb68109dd0 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb6cd05700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7feb68109dd0 0x7feb681a1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:51980/0 (socket says 172.21.15.162:51980) 2024-04-27T01:10:54.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb6dd07700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7feb6810d970 0x7feb681a5750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39804/0 (socket says 172.21.15.162:39804) 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb6cd05700 1 -- 172.21.15.162:0/1207267881 learned_addr learned my addr 172.21.15.162:0/1207267881 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3443863257 0 0) 0x7feb681a7090 con 0x7feb6810d970 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb540031f0 con 0x7feb6810d970 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1535904829 0 0) 0x7feb681a94f0 con 0x7feb68109dd0 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb681a7090 con 0x7feb68109dd0 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429060078 0 0) 0x7feb681a82b0 con 0x7feb68101140 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.113+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb681a94f0 con 0x7feb68101140 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4011678761 0 0) 0x7feb540031f0 con 0x7feb6810d970 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb681a82b0 con 0x7feb6810d970 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7feb60001db0 con 0x7feb6810d970 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2964425582 0 0) 0x7feb681a7090 con 0x7feb68109dd0 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb540031f0 con 0x7feb68109dd0 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7feb64001830 con 0x7feb68109dd0 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1314805808 0 0) 0x7feb681a94f0 con 0x7feb68101140 2024-04-27T01:10:54.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb681a7090 con 0x7feb68101140 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7feb58001880 con 0x7feb68101140 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 736881546 0 0) 0x7feb681a82b0 con 0x7feb6810d970 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 >> v1:172.21.15.162:6790/0 conn(0x7feb68109dd0 legacy=0x7feb681a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 >> v1:172.21.15.181:6789/0 conn(0x7feb68101140 legacy=0x7feb68106450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb681aa730 con 0x7feb6810d970 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb6f76a700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feb681a72c0 con 0x7feb6810d970 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb6f76a700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7feb681a7780 con 0x7feb6810d970 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7feb60002060 con 0x7feb6810d970 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.114+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7feb60005ef0 con 0x7feb6810d970 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.115+0000 7feb6f76a700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb4c005120 con 0x7feb6810d970 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.115+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7feb6001c6f0 con 0x7feb6810d970 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.116+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7feb600896b0 con 0x7feb6810d970 2024-04-27T01:10:54.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.121+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7feb60056db0 con 0x7feb6810d970 2024-04-27T01:10:54.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.322+0000 7feb6f76a700 1 -- 172.21.15.162:0/1207267881 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7feb4c005410 con 0x7feb6810d970 2024-04-27T01:10:54.313 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.323+0000 7feb5e7fc700 1 -- 172.21.15.162:0/1207267881 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (unknown 1274345170 0 2830438874) 0x7feb6005aa60 con 0x7feb6810d970 2024-04-27T01:10:54.313 INFO:teuthology.orchestra.run.smithi162.stdout:201863462933 2024-04-27T01:10:54.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.325+0000 7feb6f76a700 1 -- 172.21.15.162:0/1207267881 >> v1:172.21.15.162:6800/624703537 conn(0x7feb5406eed0 legacy=0x7feb54071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:54.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.325+0000 7feb6f76a700 1 -- 172.21.15.162:0/1207267881 >> v1:172.21.15.162:6789/0 conn(0x7feb6810d970 legacy=0x7feb681a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:54.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.325+0000 7feb6f76a700 1 -- 172.21.15.162:0/1207267881 shutdown_connections 2024-04-27T01:10:54.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.325+0000 7feb6f76a700 1 -- 172.21.15.162:0/1207267881 >> 172.21.15.162:0/1207267881 conn(0x7feb680fca10 msgr2=0x7feb68069be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:54.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.325+0000 7feb6f76a700 1 -- 172.21.15.162:0/1207267881 shutdown_connections 2024-04-27T01:10:54.315 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:54.325+0000 7feb6f76a700 1 -- 172.21.15.162:0/1207267881 wait complete. 2024-04-27T01:10:55.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:54 smithi181 ceph-mon[30237]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:10:55.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1207267881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-27T01:10:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:54 smithi162 ceph-mon[37680]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:10:55.048 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1207267881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-27T01:10:55.048 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:54 smithi162 ceph-mon[26714]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:10:55.049 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1207267881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-27T01:10:55.690 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462931 got 201863462933 for osd.6 2024-04-27T01:10:55.691 DEBUG:teuthology.parallel:result is None 2024-04-27T01:10:56.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:56 smithi162 ceph-mon[37680]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:10:56.916 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:56 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:56.916 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:56 smithi162 ceph-mon[26714]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:10:56.916 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:56 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:56 smithi181 ceph-mon[30237]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:10:57.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:56 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.395+0000 7fd301677700 1 -- 172.21.15.162:0/1384705538 >> v1:172.21.15.162:6790/0 conn(0x7fd2fc10afb0 legacy=0x7fd2fc10d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:57.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.395+0000 7fd301677700 1 -- 172.21.15.162:0/1384705538 shutdown_connections 2024-04-27T01:10:57.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.395+0000 7fd301677700 1 -- 172.21.15.162:0/1384705538 >> 172.21.15.162:0/1384705538 conn(0x7fd2fc0fca10 msgr2=0x7fd2fc0fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:57.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.395+0000 7fd301677700 1 -- 172.21.15.162:0/1384705538 shutdown_connections 2024-04-27T01:10:57.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.395+0000 7fd301677700 1 -- 172.21.15.162:0/1384705538 wait complete. 2024-04-27T01:10:57.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.396+0000 7fd301677700 1 Processor -- start 2024-04-27T01:10:57.398 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.396+0000 7fd301677700 1 -- start start 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd301677700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2fc1a9120 con 0x7fd2fc10afb0 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd301677700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2fc1aa300 con 0x7fd2fc1031a0 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd301677700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd2fc1ab540 con 0x7fd2fc10eac0 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2fb7fe700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fd2fc10eac0 0x7fd2fc1a7a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:45430/0 (socket says 172.21.15.162:45430) 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2fb7fe700 1 -- 172.21.15.162:0/3596017594 learned_addr learned my addr 172.21.15.162:0/3596017594 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2fa7fc700 1 --1- 172.21.15.162:0/3596017594 >> v1:172.21.15.162:6789/0 conn(0x7fd2fc10afb0 0x7fd2fc1a42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:46738/0 (socket says 172.21.15.162:46738) 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728313391 0 0) 0x7fd2fc1aa300 con 0x7fd2fc1031a0 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2e40035b0 con 0x7fd2fc1031a0 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3371654047 0 0) 0x7fd2fc1a9120 con 0x7fd2fc10afb0 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2fc1aa300 con 0x7fd2fc10afb0 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3287043692 0 0) 0x7fd2fc1ab540 con 0x7fd2fc10eac0 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2fc1a9120 con 0x7fd2fc10eac0 2024-04-27T01:10:57.399 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4105587100 0 0) 0x7fd2e40035b0 con 0x7fd2fc1031a0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2fc1ab540 con 0x7fd2fc1031a0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd2ec001870 con 0x7fd2fc1031a0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 90202117 0 0) 0x7fd2fc1a9120 con 0x7fd2fc10eac0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2e40035b0 con 0x7fd2fc10eac0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2404487201 0 0) 0x7fd2fc1aa300 con 0x7fd2fc10afb0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd2fc1a9120 con 0x7fd2fc10afb0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.397+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd2f4001c00 con 0x7fd2fc10eac0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.398+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd2f0004430 con 0x7fd2fc10afb0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.398+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.1 v1:172.21.15.181:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2648839287 0 0) 0x7fd2fc1ab540 con 0x7fd2fc1031a0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.398+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 >> v1:172.21.15.162:6790/0 conn(0x7fd2fc10eac0 legacy=0x7fd2fc1a7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.398+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 >> v1:172.21.15.162:6789/0 conn(0x7fd2fc10afb0 legacy=0x7fd2fc1a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.398+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.181:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc1ac780 con 0x7fd2fc1031a0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.398+0000 7fd301677700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.181:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd2fc1aa530 con 0x7fd2fc1031a0 2024-04-27T01:10:57.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.398+0000 7fd301677700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.181:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd2fc1aa9f0 con 0x7fd2fc1031a0 2024-04-27T01:10:57.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.398+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.1 v1:172.21.15.181:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fd2ec003b10 con 0x7fd2fc1031a0 2024-04-27T01:10:57.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.398+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.1 v1:172.21.15.181:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd2ec004a20 con 0x7fd2fc1031a0 2024-04-27T01:10:57.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.399+0000 7fd301677700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e8005150 con 0x7fd2fc1031a0 2024-04-27T01:10:57.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.399+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.1 v1:172.21.15.181:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7fd2ec01b4a0 con 0x7fd2fc1031a0 2024-04-27T01:10:57.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.400+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.1 v1:172.21.15.181:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7fd2ec088200 con 0x7fd2fc1031a0 2024-04-27T01:10:57.401 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.405+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.1 v1:172.21.15.181:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fd2ec055900 con 0x7fd2fc1031a0 2024-04-27T01:10:57.650 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.660+0000 7fd301677700 1 -- 172.21.15.162:0/3596017594 --> v1:172.21.15.181:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd2e8005440 con 0x7fd2fc1031a0 2024-04-27T01:10:57.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.661+0000 7fd2e3fff700 1 -- 172.21.15.162:0/3596017594 <== mon.1 v1:172.21.15.181:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 2087499782) 0x7fd2ec0595b0 con 0x7fd2fc1031a0 2024-04-27T01:10:57.651 INFO:teuthology.orchestra.run.smithi162.stdout:34359738419 2024-04-27T01:10:57.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.663+0000 7fd301677700 1 -- 172.21.15.162:0/3596017594 >> v1:172.21.15.162:6800/624703537 conn(0x7fd2e406f1b0 legacy=0x7fd2e4071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:57.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.663+0000 7fd301677700 1 -- 172.21.15.162:0/3596017594 >> v1:172.21.15.181:6789/0 conn(0x7fd2fc1031a0 legacy=0x7fd2fc10a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:57.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.663+0000 7fd301677700 1 -- 172.21.15.162:0/3596017594 shutdown_connections 2024-04-27T01:10:57.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.663+0000 7fd301677700 1 -- 172.21.15.162:0/3596017594 >> 172.21.15.162:0/3596017594 conn(0x7fd2fc0fca10 msgr2=0x7fd2fc110f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:57.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.663+0000 7fd301677700 1 -- 172.21.15.162:0/3596017594 shutdown_connections 2024-04-27T01:10:57.653 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:57.663+0000 7fd301677700 1 -- 172.21.15.162:0/3596017594 wait complete. 2024-04-27T01:10:57.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.978 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi162 to 3760M 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: Updating smithi162:/etc/ceph/ceph.conf 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: Updating smithi181:/etc/ceph/ceph.conf 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:10:57.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3596017594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.980 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi162 to 3760M 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: Updating smithi162:/etc/ceph/ceph.conf 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: Updating smithi181:/etc/ceph/ceph.conf 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3596017594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T01:10:57.981 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:57.982 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:57 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi162 to 3760M 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: Updating smithi162:/etc/ceph/ceph.conf 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: Updating smithi181:/etc/ceph/ceph.conf 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:58.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:58.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3596017594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-27T01:10:58.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:58.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:57 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:10:58.643 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.653+0000 7ff3e0f40700 1 -- 172.21.15.162:0/574690488 >> v1:172.21.15.162:6789/0 conn(0x7ff3dc1031a0 legacy=0x7ff3dc103580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:58.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.654+0000 7ff3e0f40700 1 -- 172.21.15.162:0/574690488 shutdown_connections 2024-04-27T01:10:58.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.654+0000 7ff3e0f40700 1 -- 172.21.15.162:0/574690488 >> 172.21.15.162:0/574690488 conn(0x7ff3dc0fca10 msgr2=0x7ff3dc0fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:58.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.654+0000 7ff3e0f40700 1 -- 172.21.15.162:0/574690488 shutdown_connections 2024-04-27T01:10:58.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.654+0000 7ff3e0f40700 1 -- 172.21.15.162:0/574690488 wait complete. 2024-04-27T01:10:58.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.654+0000 7ff3e0f40700 1 Processor -- start 2024-04-27T01:10:58.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.654+0000 7ff3e0f40700 1 -- start start 2024-04-27T01:10:58.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3e0f40700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff3dc1a70e0 con 0x7ff3dc1031a0 2024-04-27T01:10:58.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3e0f40700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff3dc1a8300 con 0x7ff3dc10afb0 2024-04-27T01:10:58.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3e0f40700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7ff3dc1a9540 con 0x7ff3dc10eac0 2024-04-27T01:10:58.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3da59c700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7ff3dc1031a0 0x7ff3dc10a800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:46762/0 (socket says 172.21.15.162:46762) 2024-04-27T01:10:58.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3da59c700 1 -- 172.21.15.162:0/4002708203 learned_addr learned my addr 172.21.15.162:0/4002708203 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:58.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 825896823 0 0) 0x7ff3dc1a70e0 con 0x7ff3dc1031a0 2024-04-27T01:10:58.646 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3c40031f0 con 0x7ff3dc1031a0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1596043998 0 0) 0x7ff3dc1a9540 con 0x7ff3dc10eac0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3dc1a70e0 con 0x7ff3dc10eac0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 75069404 0 0) 0x7ff3dc1a70e0 con 0x7ff3dc10eac0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3dc1a9540 con 0x7ff3dc10eac0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1509504087 0 0) 0x7ff3c40031f0 con 0x7ff3dc1031a0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3dc1a70e0 con 0x7ff3dc1031a0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7ff3d4001590 con 0x7ff3dc10eac0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7ff3cc004510 con 0x7ff3dc1031a0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4147783904 0 0) 0x7ff3dc1a8300 con 0x7ff3dc10afb0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3c40031f0 con 0x7ff3dc10afb0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 838295481 0 0) 0x7ff3dc1a9540 con 0x7ff3dc10eac0 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 >> v1:172.21.15.181:6789/0 conn(0x7ff3dc10afb0 legacy=0x7ff3dc1a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 >> v1:172.21.15.162:6789/0 conn(0x7ff3dc1031a0 legacy=0x7ff3dc10a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:58.647 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3dc1aa780 con 0x7ff3dc10eac0 2024-04-27T01:10:58.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.655+0000 7ff3e0f40700 1 -- 172.21.15.162:0/4002708203 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff3dc1a9770 con 0x7ff3dc10eac0 2024-04-27T01:10:58.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.656+0000 7ff3e0f40700 1 -- 172.21.15.162:0/4002708203 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff3dc1a9c30 con 0x7ff3dc10eac0 2024-04-27T01:10:58.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.656+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7ff3d4003e90 con 0x7ff3dc10eac0 2024-04-27T01:10:58.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.656+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7ff3d4004d50 con 0x7ff3dc10eac0 2024-04-27T01:10:58.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.657+0000 7ff3e0f40700 1 -- 172.21.15.162:0/4002708203 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3bc005120 con 0x7ff3dc10eac0 2024-04-27T01:10:58.648 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.657+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7ff3d4003a00 con 0x7ff3dc10eac0 2024-04-27T01:10:58.649 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.658+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7ff3d4088690 con 0x7ff3dc10eac0 2024-04-27T01:10:58.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.662+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7ff3d4055d90 con 0x7ff3dc10eac0 2024-04-27T01:10:58.860 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.870+0000 7ff3e0f40700 1 -- 172.21.15.162:0/4002708203 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff3bc005410 con 0x7ff3dc10eac0 2024-04-27T01:10:58.861 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.871+0000 7ff3cb7fe700 1 -- 172.21.15.162:0/4002708203 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (unknown 1482059429 0 68472687) 0x7ff3d4059a40 con 0x7ff3dc10eac0 2024-04-27T01:10:58.861 INFO:teuthology.orchestra.run.smithi162.stdout:236223201298 2024-04-27T01:10:58.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.873+0000 7ff3e0f40700 1 -- 172.21.15.162:0/4002708203 >> v1:172.21.15.162:6800/624703537 conn(0x7ff3c406eee0 legacy=0x7ff3c40713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:58.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.873+0000 7ff3e0f40700 1 -- 172.21.15.162:0/4002708203 >> v1:172.21.15.162:6790/0 conn(0x7ff3dc10eac0 legacy=0x7ff3dc1a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:58.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.873+0000 7ff3e0f40700 1 -- 172.21.15.162:0/4002708203 shutdown_connections 2024-04-27T01:10:58.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.873+0000 7ff3e0f40700 1 -- 172.21.15.162:0/4002708203 >> 172.21.15.162:0/4002708203 conn(0x7ff3dc0fca10 msgr2=0x7ff3dc110f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:58.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.873+0000 7ff3e0f40700 1 -- 172.21.15.162:0/4002708203 shutdown_connections 2024-04-27T01:10:58.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:58.873+0000 7ff3e0f40700 1 -- 172.21.15.162:0/4002708203 wait complete. 2024-04-27T01:10:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[37680]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:10:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[37680]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:10:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[37680]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:10:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[37680]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:10:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[37680]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:10:58.887 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[37680]: Deploying daemon alertmanager.a on smithi162 2024-04-27T01:10:58.887 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[26714]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:10:58.887 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[26714]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:10:58.888 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[26714]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:10:58.888 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[26714]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:10:58.888 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[26714]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:10:58.888 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:58 smithi162 ceph-mon[26714]: Deploying daemon alertmanager.a on smithi162 2024-04-27T01:10:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:58 smithi181 ceph-mon[30237]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:10:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:58 smithi181 ceph-mon[30237]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:10:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:58 smithi181 ceph-mon[30237]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2024-04-27T01:10:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:58 smithi181 ceph-mon[30237]: Updating smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:10:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:58 smithi181 ceph-mon[30237]: Updating smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-27T01:10:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:58 smithi181 ceph-mon[30237]: Deploying daemon alertmanager.a on smithi162 2024-04-27T01:10:59.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.098+0000 7f8d3039b700 1 -- 172.21.15.162:0/2172383736 >> v1:172.21.15.162:6790/0 conn(0x7f8d281011b0 legacy=0x7f8d28101590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:59.088 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.098+0000 7f8d3039b700 1 -- 172.21.15.162:0/2172383736 shutdown_connections 2024-04-27T01:10:59.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.098+0000 7f8d3039b700 1 -- 172.21.15.162:0/2172383736 >> 172.21.15.162:0/2172383736 conn(0x7f8d280fca40 msgr2=0x7f8d280fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:59.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.098+0000 7f8d3039b700 1 -- 172.21.15.162:0/2172383736 shutdown_connections 2024-04-27T01:10:59.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.098+0000 7f8d3039b700 1 -- 172.21.15.162:0/2172383736 wait complete. 2024-04-27T01:10:59.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.098+0000 7f8d3039b700 1 Processor -- start 2024-04-27T01:10:59.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.098+0000 7f8d3039b700 1 -- start start 2024-04-27T01:10:59.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d3039b700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8d2810fc40 con 0x7f8d2810daf0 2024-04-27T01:10:59.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d3039b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8d2810fe50 con 0x7f8d28109f50 2024-04-27T01:10:59.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d3039b700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8d281abad0 con 0x7f8d281011b0 2024-04-27T01:10:59.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d2e137700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f8d281011b0 0x7f8d2810e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:45470/0 (socket says 172.21.15.162:45470) 2024-04-27T01:10:59.089 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d2e137700 1 -- 172.21.15.162:0/3183199437 learned_addr learned my addr 172.21.15.162:0/3183199437 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:10:59.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518016228 0 0) 0x7f8d2810fc40 con 0x7f8d2810daf0 2024-04-27T01:10:59.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d140035b0 con 0x7f8d2810daf0 2024-04-27T01:10:59.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4190708697 0 0) 0x7f8d281abad0 con 0x7f8d281011b0 2024-04-27T01:10:59.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d2810fc40 con 0x7f8d281011b0 2024-04-27T01:10:59.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 60927971 0 0) 0x7f8d2810fe50 con 0x7f8d28109f50 2024-04-27T01:10:59.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d281abad0 con 0x7f8d28109f50 2024-04-27T01:10:59.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2873857978 0 0) 0x7f8d140035b0 con 0x7f8d2810daf0 2024-04-27T01:10:59.091 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d2810fe50 con 0x7f8d2810daf0 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8d20001ae0 con 0x7f8d2810daf0 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1248516871 0 0) 0x7f8d2810fc40 con 0x7f8d281011b0 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d140035b0 con 0x7f8d281011b0 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8d240018c0 con 0x7f8d281011b0 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 765871765 0 0) 0x7f8d281abad0 con 0x7f8d28109f50 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d2810fc40 con 0x7f8d28109f50 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.099+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8d18001800 con 0x7f8d28109f50 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.100+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4206863552 0 0) 0x7f8d2810fe50 con 0x7f8d2810daf0 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.100+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 >> v1:172.21.15.162:6790/0 conn(0x7f8d281011b0 legacy=0x7f8d2810e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.100+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 >> v1:172.21.15.181:6789/0 conn(0x7f8d28109f50 legacy=0x7f8d2810e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.100+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d281accb0 con 0x7f8d2810daf0 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.100+0000 7f8d3039b700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8d281abd00 con 0x7f8d2810daf0 2024-04-27T01:10:59.092 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.100+0000 7f8d3039b700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8d281ac1c0 con 0x7f8d2810daf0 2024-04-27T01:10:59.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.100+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f8d20003e90 con 0x7f8d2810daf0 2024-04-27T01:10:59.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.100+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8d20004d20 con 0x7f8d2810daf0 2024-04-27T01:10:59.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.101+0000 7f8d3039b700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d0c005120 con 0x7f8d2810daf0 2024-04-27T01:10:59.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.101+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f8d2001b520 con 0x7f8d2810daf0 2024-04-27T01:10:59.093 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.102+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7f8d20088640 con 0x7f8d2810daf0 2024-04-27T01:10:59.096 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.106+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f8d20055da0 con 0x7f8d2810daf0 2024-04-27T01:10:59.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.313+0000 7f8d3039b700 1 -- 172.21.15.162:0/3183199437 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f8d0c005410 con 0x7f8d2810daf0 2024-04-27T01:10:59.303 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.313+0000 7f8d1f7fe700 1 -- 172.21.15.162:0/3183199437 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (unknown 1823589948 0 1544327744) 0x7f8d20059a50 con 0x7f8d2810daf0 2024-04-27T01:10:59.303 INFO:teuthology.orchestra.run.smithi162.stdout:133143986208 2024-04-27T01:10:59.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.315+0000 7f8d3039b700 1 -- 172.21.15.162:0/3183199437 >> v1:172.21.15.162:6800/624703537 conn(0x7f8d1406ef80 legacy=0x7f8d14071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:59.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.315+0000 7f8d3039b700 1 -- 172.21.15.162:0/3183199437 >> v1:172.21.15.162:6789/0 conn(0x7f8d2810daf0 legacy=0x7f8d281a8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:10:59.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.315+0000 7f8d3039b700 1 -- 172.21.15.162:0/3183199437 shutdown_connections 2024-04-27T01:10:59.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.315+0000 7f8d3039b700 1 -- 172.21.15.162:0/3183199437 >> 172.21.15.162:0/3183199437 conn(0x7f8d280fca40 msgr2=0x7f8d28110b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:10:59.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.315+0000 7f8d3039b700 1 -- 172.21.15.162:0/3183199437 shutdown_connections 2024-04-27T01:10:59.305 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:10:59.315+0000 7f8d3039b700 1 -- 172.21.15.162:0/3183199437 wait complete. 2024-04-27T01:10:59.871 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201296 got 236223201298 for osd.7 2024-04-27T01:10:59.871 DEBUG:teuthology.parallel:result is None 2024-04-27T01:11:00.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4002708203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-27T01:11:00.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:10:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3183199437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T01:11:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4002708203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-27T01:11:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:10:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3183199437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T01:11:00.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4002708203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-27T01:11:00.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:10:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3183199437' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-27T01:11:00.089 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:10:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:10:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:11:00.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.299+0000 7f890b2c0700 1 -- 172.21.15.162:0/1315346328 >> v1:172.21.15.162:6789/0 conn(0x7f8904102260 legacy=0x7f8904102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:00.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.299+0000 7f890b2c0700 1 -- 172.21.15.162:0/1315346328 shutdown_connections 2024-04-27T01:11:00.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.299+0000 7f890b2c0700 1 -- 172.21.15.162:0/1315346328 >> 172.21.15.162:0/1315346328 conn(0x7f89040fdb50 msgr2=0x7f89040fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:00.291 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.299+0000 7f890b2c0700 1 -- 172.21.15.162:0/1315346328 shutdown_connections 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.299+0000 7f890b2c0700 1 -- 172.21.15.162:0/1315346328 wait complete. 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.300+0000 7f890b2c0700 1 Processor -- start 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.300+0000 7f890b2c0700 1 -- start start 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.300+0000 7f890b2c0700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89041a6eb0 con 0x7f8904102260 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.300+0000 7f890b2c0700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89041a80d0 con 0x7f890410aef0 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.300+0000 7f890b2c0700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f89041a9310 con 0x7f890410ea90 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.300+0000 7f890905c700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f8904102260 0x7f8904107340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:46794/0 (socket says 172.21.15.162:46794) 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.300+0000 7f890905c700 1 -- 172.21.15.162:0/2552213979 learned_addr learned my addr 172.21.15.162:0/2552213979 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.300+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 560160886 0 0) 0x7f89041a9310 con 0x7f890410ea90 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.300+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f00035b0 con 0x7f890410ea90 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2883200299 0 0) 0x7f89041a6eb0 con 0x7f8904102260 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89041a9310 con 0x7f8904102260 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 496109718 0 0) 0x7f89041a80d0 con 0x7f890410aef0 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89041a6eb0 con 0x7f890410aef0 2024-04-27T01:11:00.292 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1308353636 0 0) 0x7f88f00035b0 con 0x7f890410ea90 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89041a80d0 con 0x7f890410ea90 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f88fc0016f0 con 0x7f890410ea90 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2272905275 0 0) 0x7f89041a9310 con 0x7f8904102260 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88f00035b0 con 0x7f8904102260 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f88f4004480 con 0x7f8904102260 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2504214341 0 0) 0x7f89041a6eb0 con 0x7f890410aef0 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89041a9310 con 0x7f890410aef0 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8900001830 con 0x7f890410aef0 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2375691485 0 0) 0x7f89041a80d0 con 0x7f890410ea90 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 >> v1:172.21.15.181:6789/0 conn(0x7f890410aef0 legacy=0x7f89041a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 >> v1:172.21.15.162:6789/0 conn(0x7f8904102260 legacy=0x7f8904107340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89041aa550 con 0x7f890410ea90 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f890b2c0700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89041a70e0 con 0x7f890410ea90 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f890b2c0700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f89041a75a0 con 0x7f890410ea90 2024-04-27T01:11:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f88fc0019a0 con 0x7f890410ea90 2024-04-27T01:11:00.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.301+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f88fc004dc0 con 0x7f890410ea90 2024-04-27T01:11:00.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.302+0000 7f890b2c0700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e8005120 con 0x7f890410ea90 2024-04-27T01:11:00.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.302+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f88fc003cb0 con 0x7f890410ea90 2024-04-27T01:11:00.294 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.303+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7f88fc0885d0 con 0x7f890410ea90 2024-04-27T01:11:00.297 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.307+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f88fc055cd0 con 0x7f890410ea90 2024-04-27T01:11:00.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.512+0000 7f890b2c0700 1 -- 172.21.15.162:0/2552213979 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f88e8005410 con 0x7f890410ea90 2024-04-27T01:11:00.502 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.512+0000 7f88fa7fc700 1 -- 172.21.15.162:0/2552213979 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (unknown 383520633 0 1028728461) 0x7f88fc059980 con 0x7f890410ea90 2024-04-27T01:11:00.502 INFO:teuthology.orchestra.run.smithi162.stdout:111669149734 2024-04-27T01:11:00.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.514+0000 7f890b2c0700 1 -- 172.21.15.162:0/2552213979 >> v1:172.21.15.162:6800/624703537 conn(0x7f88f006ef30 legacy=0x7f88f00713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:00.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.514+0000 7f890b2c0700 1 -- 172.21.15.162:0/2552213979 >> v1:172.21.15.162:6790/0 conn(0x7f890410ea90 legacy=0x7f89041a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:00.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.514+0000 7f890b2c0700 1 -- 172.21.15.162:0/2552213979 shutdown_connections 2024-04-27T01:11:00.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.514+0000 7f890b2c0700 1 -- 172.21.15.162:0/2552213979 >> 172.21.15.162:0/2552213979 conn(0x7f89040fdb50 msgr2=0x7f89040fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:00.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.514+0000 7f890b2c0700 1 -- 172.21.15.162:0/2552213979 shutdown_connections 2024-04-27T01:11:00.504 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:00.514+0000 7f890b2c0700 1 -- 172.21.15.162:0/2552213979 wait complete. 2024-04-27T01:11:00.727 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986205 got 133143986208 for osd.4 2024-04-27T01:11:00.728 DEBUG:teuthology.parallel:result is None 2024-04-27T01:11:00.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:00 smithi162 ceph-mon[26714]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:00.818 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2552213979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T01:11:00.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:00 smithi162 ceph-mon[37680]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:00.819 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2552213979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T01:11:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:00 smithi181 ceph-mon[30237]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2552213979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-27T01:11:01.735 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149730 got 111669149734 for osd.3 2024-04-27T01:11:01.735 DEBUG:teuthology.parallel:result is None 2024-04-27T01:11:01.977 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:01 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:01.977 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:01 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:02.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:01 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:02.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:11:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:11:02.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.550+0000 7fd4aaa13700 1 -- 172.21.15.162:0/2326420637 >> v1:172.21.15.162:6789/0 conn(0x7fd4a410eab0 legacy=0x7fd4a4110f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:02.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.550+0000 7fd4aaa13700 1 -- 172.21.15.162:0/2326420637 shutdown_connections 2024-04-27T01:11:02.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.550+0000 7fd4aaa13700 1 -- 172.21.15.162:0/2326420637 >> 172.21.15.162:0/2326420637 conn(0x7fd4a4076250 msgr2=0x7fd4a4076660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:02.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.550+0000 7fd4aaa13700 1 -- 172.21.15.162:0/2326420637 shutdown_connections 2024-04-27T01:11:02.540 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.550+0000 7fd4aaa13700 1 -- 172.21.15.162:0/2326420637 wait complete. 2024-04-27T01:11:02.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.550+0000 7fd4aaa13700 1 Processor -- start 2024-04-27T01:11:02.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.550+0000 7fd4aaa13700 1 -- start start 2024-04-27T01:11:02.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4aaa13700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4a41a70d0 con 0x7fd4a410af40 2024-04-27T01:11:02.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4aaa13700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4a41a82f0 con 0x7fd4a4100ad0 2024-04-27T01:11:02.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4aaa13700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd4a41a9530 con 0x7fd4a410eab0 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4a37fe700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fd4a410af40 0x7fd4a41a1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:46824/0 (socket says 172.21.15.162:46824) 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4a37fe700 1 -- 172.21.15.162:0/1534899933 learned_addr learned my addr 172.21.15.162:0/1534899933 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423058990 0 0) 0x7fd4a41a9530 con 0x7fd4a410eab0 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd48c0035b0 con 0x7fd4a410eab0 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2572784623 0 0) 0x7fd4a41a70d0 con 0x7fd4a410af40 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4a41a9530 con 0x7fd4a410af40 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1191562514 0 0) 0x7fd4a41a82f0 con 0x7fd4a4100ad0 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4a41a70d0 con 0x7fd4a4100ad0 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.551+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3861456925 0 0) 0x7fd48c0035b0 con 0x7fd4a410eab0 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4a41a82f0 con 0x7fd4a410eab0 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 334652801 0 0) 0x7fd4a41a9530 con 0x7fd4a410af40 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd48c0035b0 con 0x7fd4a410af40 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd49c001d60 con 0x7fd4a410eab0 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd498001770 con 0x7fd4a410af40 2024-04-27T01:11:02.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 467581144 0 0) 0x7fd4a41a70d0 con 0x7fd4a4100ad0 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd4a41a9530 con 0x7fd4a4100ad0 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd4940013b0 con 0x7fd4a4100ad0 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 94696980 0 0) 0x7fd4a41a82f0 con 0x7fd4a410eab0 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 >> v1:172.21.15.181:6789/0 conn(0x7fd4a4100ad0 legacy=0x7fd4a4107550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 >> v1:172.21.15.162:6789/0 conn(0x7fd4a410af40 legacy=0x7fd4a41a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a41aa770 con 0x7fd4a410eab0 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4aaa13700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a41a7300 con 0x7fd4a410eab0 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4aaa13700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd4a41a77c0 con 0x7fd4a410eab0 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fd49c001570 con 0x7fd4a410eab0 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.552+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd49c004f50 con 0x7fd4a410eab0 2024-04-27T01:11:02.544 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.553+0000 7fd4aaa13700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd490005120 con 0x7fd4a410eab0 2024-04-27T01:11:02.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.553+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7fd49c01b9d0 con 0x7fd4a410eab0 2024-04-27T01:11:02.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.554+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7fd49c0887b0 con 0x7fd4a410eab0 2024-04-27T01:11:02.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.558+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fd49c004a20 con 0x7fd4a410eab0 2024-04-27T01:11:02.565 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:11:02.758 INFO:teuthology.orchestra.run.smithi162.stdout:55834574896 2024-04-27T01:11:02.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.768+0000 7fd4aaa13700 1 -- 172.21.15.162:0/1534899933 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd490005410 con 0x7fd4a410eab0 2024-04-27T01:11:02.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.768+0000 7fd4a17fa700 1 -- 172.21.15.162:0/1534899933 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 1580136542) 0x7fd49c05e420 con 0x7fd4a410eab0 2024-04-27T01:11:02.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.770+0000 7fd4aaa13700 1 -- 172.21.15.162:0/1534899933 >> v1:172.21.15.162:6800/624703537 conn(0x7fd48c06f020 legacy=0x7fd48c0714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:02.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.770+0000 7fd4aaa13700 1 -- 172.21.15.162:0/1534899933 >> v1:172.21.15.162:6790/0 conn(0x7fd4a410eab0 legacy=0x7fd4a41a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:02.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.770+0000 7fd4aaa13700 1 -- 172.21.15.162:0/1534899933 shutdown_connections 2024-04-27T01:11:02.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.770+0000 7fd4aaa13700 1 -- 172.21.15.162:0/1534899933 >> 172.21.15.162:0/1534899933 conn(0x7fd4a4076250 msgr2=0x7fd4a4112220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:02.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.770+0000 7fd4aaa13700 1 -- 172.21.15.162:0/1534899933 shutdown_connections 2024-04-27T01:11:02.761 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:02.770+0000 7fd4aaa13700 1 -- 172.21.15.162:0/1534899933 wait complete. 2024-04-27T01:11:03.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:02 smithi181 ceph-mon[30237]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-27T01:11:03.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:02 smithi162 ceph-mon[26714]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-27T01:11:03.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:02 smithi162 ceph-mon[37680]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-27T01:11:03.204 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738419 for osd.0 2024-04-27T01:11:03.205 DEBUG:teuthology.parallel:result is None 2024-04-27T01:11:03.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:03.784 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1534899933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T01:11:03.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:03.842 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1534899933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T01:11:03.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:03.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1534899933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-27T01:11:04.152 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574896 for osd.1 2024-04-27T01:11:04.152 DEBUG:teuthology.parallel:result is None 2024-04-27T01:11:04.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.348+0000 7fcb1f13e700 1 -- 172.21.15.162:0/755090128 >> v1:172.21.15.162:6789/0 conn(0x7fcb18101160 legacy=0x7fcb18101540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:04.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.348+0000 7fcb1f13e700 1 -- 172.21.15.162:0/755090128 shutdown_connections 2024-04-27T01:11:04.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.348+0000 7fcb1f13e700 1 -- 172.21.15.162:0/755090128 >> 172.21.15.162:0/755090128 conn(0x7fcb180fca10 msgr2=0x7fcb180fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:04.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.348+0000 7fcb1f13e700 1 -- 172.21.15.162:0/755090128 shutdown_connections 2024-04-27T01:11:04.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.348+0000 7fcb1f13e700 1 -- 172.21.15.162:0/755090128 wait complete. 2024-04-27T01:11:04.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.348+0000 7fcb1f13e700 1 Processor -- start 2024-04-27T01:11:04.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.348+0000 7fcb1f13e700 1 -- start start 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb1f13e700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcb181a9330 con 0x7fcb1810daa0 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb1f13e700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcb181aa550 con 0x7fcb18101160 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb1f13e700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fcb181ab790 con 0x7fcb18109f00 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0ffff700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fcb18109f00 0x7fcb181a4310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:45518/0 (socket says 172.21.15.162:45518) 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0ffff700 1 -- 172.21.15.162:0/4171526156 learned_addr learned my addr 172.21.15.162:0/4171526156 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb1d6db700 1 --1- 172.21.15.162:0/4171526156 >> v1:172.21.15.162:6789/0 conn(0x7fcb1810daa0 0x7fcb181a7a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:46842/0 (socket says 172.21.15.162:46842) 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2149120365 0 0) 0x7fcb181a9330 con 0x7fcb1810daa0 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb000035b0 con 0x7fcb1810daa0 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 28052588 0 0) 0x7fcb181ab790 con 0x7fcb18109f00 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb181a9330 con 0x7fcb18109f00 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3868594500 0 0) 0x7fcb181aa550 con 0x7fcb18101160 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb181ab790 con 0x7fcb18101160 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1111452727 0 0) 0x7fcb000035b0 con 0x7fcb1810daa0 2024-04-27T01:11:04.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb181aa550 con 0x7fcb1810daa0 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fcb10001cc0 con 0x7fcb1810daa0 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3639272349 0 0) 0x7fcb181a9330 con 0x7fcb18109f00 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb000035b0 con 0x7fcb18109f00 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3244789063 0 0) 0x7fcb181ab790 con 0x7fcb18101160 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb181a9330 con 0x7fcb18101160 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fcb08001870 con 0x7fcb18109f00 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fcb14001840 con 0x7fcb18101160 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 33694234 0 0) 0x7fcb181aa550 con 0x7fcb1810daa0 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.349+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 >> v1:172.21.15.162:6790/0 conn(0x7fcb18109f00 legacy=0x7fcb181a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.350+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 >> v1:172.21.15.181:6789/0 conn(0x7fcb18101160 legacy=0x7fcb181063d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.350+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb181ac9d0 con 0x7fcb1810daa0 2024-04-27T01:11:04.342 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.350+0000 7fcb1f13e700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcb181ab9c0 con 0x7fcb1810daa0 2024-04-27T01:11:04.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.350+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fcb10003e90 con 0x7fcb1810daa0 2024-04-27T01:11:04.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.350+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fcb10005f90 con 0x7fcb1810daa0 2024-04-27T01:11:04.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.350+0000 7fcb1f13e700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcb181abe80 con 0x7fcb1810daa0 2024-04-27T01:11:04.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.351+0000 7fcb1f13e700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb18102640 con 0x7fcb1810daa0 2024-04-27T01:11:04.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.351+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7fcb10003a00 con 0x7fcb1810daa0 2024-04-27T01:11:04.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.352+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7fcb10089b10 con 0x7fcb1810daa0 2024-04-27T01:11:04.345 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.356+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fcb10057210 con 0x7fcb1810daa0 2024-04-27T01:11:04.554 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.565+0000 7fcb1f13e700 1 -- 172.21.15.162:0/4171526156 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fcb18068b50 con 0x7fcb1810daa0 2024-04-27T01:11:04.555 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.565+0000 7fcb0dffb700 1 -- 172.21.15.162:0/4171526156 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (unknown 2131975755 0 616938113) 0x7fcb1005aec0 con 0x7fcb1810daa0 2024-04-27T01:11:04.555 INFO:teuthology.orchestra.run.smithi162.stdout:167503724572 2024-04-27T01:11:04.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.567+0000 7fcb1f13e700 1 -- 172.21.15.162:0/4171526156 >> v1:172.21.15.162:6800/624703537 conn(0x7fcb0006f060 legacy=0x7fcb00071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:04.557 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.567+0000 7fcb1f13e700 1 -- 172.21.15.162:0/4171526156 >> v1:172.21.15.162:6789/0 conn(0x7fcb1810daa0 legacy=0x7fcb181a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:04.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.568+0000 7fcb1f13e700 1 -- 172.21.15.162:0/4171526156 shutdown_connections 2024-04-27T01:11:04.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.568+0000 7fcb1f13e700 1 -- 172.21.15.162:0/4171526156 >> 172.21.15.162:0/4171526156 conn(0x7fcb180fca10 msgr2=0x7fcb180fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:04.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.568+0000 7fcb1f13e700 1 -- 172.21.15.162:0/4171526156 shutdown_connections 2024-04-27T01:11:04.558 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:04.568+0000 7fcb1f13e700 1 -- 172.21.15.162:0/4171526156 wait complete. 2024-04-27T01:11:05.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:04 smithi181 ceph-mon[30237]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-27T01:11:05.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4171526156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T01:11:05.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:04 smithi162 ceph-mon[37680]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-27T01:11:05.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4171526156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T01:11:05.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:04 smithi162 ceph-mon[26714]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-27T01:11:05.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4171526156' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-27T01:11:05.503 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724567 got 167503724572 for osd.5 2024-04-27T01:11:05.503 DEBUG:teuthology.parallel:result is None 2024-04-27T01:11:06.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.410+0000 7f3f7206b700 1 -- 172.21.15.162:0/3212679086 >> v1:172.21.15.162:6789/0 conn(0x7f3f6c10af30 legacy=0x7f3f6c10d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:06.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.410+0000 7f3f7206b700 1 -- 172.21.15.162:0/3212679086 shutdown_connections 2024-04-27T01:11:06.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.410+0000 7f3f7206b700 1 -- 172.21.15.162:0/3212679086 >> 172.21.15.162:0/3212679086 conn(0x7f3f6c0fdb50 msgr2=0x7f3f6c0fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:06.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.410+0000 7f3f7206b700 1 -- 172.21.15.162:0/3212679086 shutdown_connections 2024-04-27T01:11:06.400 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.410+0000 7f3f7206b700 1 -- 172.21.15.162:0/3212679086 wait complete. 2024-04-27T01:11:06.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.411+0000 7f3f7206b700 1 Processor -- start 2024-04-27T01:11:06.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.411+0000 7f3f7206b700 1 -- start start 2024-04-27T01:11:06.402 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.411+0000 7f3f7206b700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f6c1a7070 con 0x7f3f6c10ead0 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.411+0000 7f3f7206b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f6c1a8290 con 0x7f3f6c10af30 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.411+0000 7f3f7206b700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3f6c1a94d0 con 0x7f3f6c1022a0 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.411+0000 7f3f6b7fe700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f3f6c1022a0 0x7f3f6c107510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:53204/0 (socket says 172.21.15.162:53204) 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.411+0000 7f3f6b7fe700 1 -- 172.21.15.162:0/1330277861 learned_addr learned my addr 172.21.15.162:0/1330277861 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.411+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2605605650 0 0) 0x7f3f6c1a7070 con 0x7f3f6c10ead0 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.411+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f540031f0 con 0x7f3f6c10ead0 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4086774263 0 0) 0x7f3f6c1a94d0 con 0x7f3f6c1022a0 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f6c1a7070 con 0x7f3f6c1022a0 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3260819416 0 0) 0x7f3f6c1a8290 con 0x7f3f6c10af30 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f6c1a94d0 con 0x7f3f6c10af30 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1047980765 0 0) 0x7f3f540031f0 con 0x7f3f6c10ead0 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f6c1a8290 con 0x7f3f6c10ead0 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f3f64001b70 con 0x7f3f6c10ead0 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 523473913 0 0) 0x7f3f6c1a7070 con 0x7f3f6c1022a0 2024-04-27T01:11:06.403 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f540031f0 con 0x7f3f6c1022a0 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f3f60001230 con 0x7f3f6c1022a0 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3761100543 0 0) 0x7f3f6c1a94d0 con 0x7f3f6c10af30 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f6c1a7070 con 0x7f3f6c10af30 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f3f5c004440 con 0x7f3f6c10af30 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2587494267 0 0) 0x7f3f6c1a8290 con 0x7f3f6c10ead0 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 >> v1:172.21.15.162:6790/0 conn(0x7f3f6c1022a0 legacy=0x7f3f6c107510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 >> v1:172.21.15.181:6789/0 conn(0x7f3f6c10af30 legacy=0x7f3f6c1a1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6c1aa710 con 0x7f3f6c10ead0 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f7206b700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f6c1a9700 con 0x7f3f6c10ead0 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.412+0000 7f3f7206b700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f6c1a9bc0 con 0x7f3f6c10ead0 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.413+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f3f64003e90 con 0x7f3f6c10ead0 2024-04-27T01:11:06.404 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.413+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f3f64004db0 con 0x7f3f6c10ead0 2024-04-27T01:11:06.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.413+0000 7f3f7206b700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f58005120 con 0x7f3f6c10ead0 2024-04-27T01:11:06.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.414+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f3f6401b590 con 0x7f3f6c10ead0 2024-04-27T01:11:06.405 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.414+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7f3f64088760 con 0x7f3f6c10ead0 2024-04-27T01:11:06.408 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.419+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f3f64055ec0 con 0x7f3f6c10ead0 2024-04-27T01:11:06.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.621+0000 7f3f7206b700 1 -- 172.21.15.162:0/1330277861 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f3f58005410 con 0x7f3f6c10ead0 2024-04-27T01:11:06.611 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.622+0000 7f3f68ff9700 1 -- 172.21.15.162:0/1330277861 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 3499689557) 0x7f3f64059b70 con 0x7f3f6c10ead0 2024-04-27T01:11:06.611 INFO:teuthology.orchestra.run.smithi162.stdout:77309411372 2024-04-27T01:11:06.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.624+0000 7f3f7206b700 1 -- 172.21.15.162:0/1330277861 >> v1:172.21.15.162:6800/624703537 conn(0x7f3f5406f020 legacy=0x7f3f540714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:06.613 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.624+0000 7f3f7206b700 1 -- 172.21.15.162:0/1330277861 >> v1:172.21.15.162:6789/0 conn(0x7f3f6c10ead0 legacy=0x7f3f6c1a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:06.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.624+0000 7f3f7206b700 1 -- 172.21.15.162:0/1330277861 shutdown_connections 2024-04-27T01:11:06.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.624+0000 7f3f7206b700 1 -- 172.21.15.162:0/1330277861 >> 172.21.15.162:0/1330277861 conn(0x7f3f6c0fdb50 msgr2=0x7f3f6c107e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:06.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.624+0000 7f3f7206b700 1 -- 172.21.15.162:0/1330277861 shutdown_connections 2024-04-27T01:11:06.614 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:06.624+0000 7f3f7206b700 1 -- 172.21.15.162:0/1330277861 wait complete. 2024-04-27T01:11:07.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:06 smithi181 ceph-mon[30237]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:07.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1330277861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T01:11:07.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:06 smithi162 ceph-mon[37680]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:07.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1330277861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T01:11:07.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:06 smithi162 ceph-mon[26714]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:07.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1330277861' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-27T01:11:07.619 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411372 for osd.2 2024-04-27T01:11:07.619 DEBUG:teuthology.parallel:result is None 2024-04-27T01:11:07.619 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-27T01:11:07.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T01:11:07.898 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:11:09.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:08 smithi181 ceph-mon[30237]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:09.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:08 smithi162 ceph-mon[37680]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:09.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:08 smithi162 ceph-mon[26714]: pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:09.958 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:11:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:11:09.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.967+0000 7f78cb868700 1 -- 172.21.15.162:0/617466813 >> v1:172.21.15.162:6790/0 conn(0x7f78c410d980 legacy=0x7f78c410fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:09.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.968+0000 7f78cb868700 1 -- 172.21.15.162:0/617466813 shutdown_connections 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.968+0000 7f78cb868700 1 -- 172.21.15.162:0/617466813 >> 172.21.15.162:0/617466813 conn(0x7f78c40fca00 msgr2=0x7f78c40fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.968+0000 7f78cb868700 1 -- 172.21.15.162:0/617466813 shutdown_connections 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.968+0000 7f78cb868700 1 -- 172.21.15.162:0/617466813 wait complete. 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.968+0000 7f78cb868700 1 Processor -- start 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.968+0000 7f78cb868700 1 -- start start 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78cb868700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78c41068b0 con 0x7f78c4101150 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78cb868700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78c41a8560 con 0x7f78c410d980 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78cb868700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f78c41a9740 con 0x7f78c4109de0 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78c8e03700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f78c4109de0 0x7f78c4104d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:53232/0 (socket says 172.21.15.162:53232) 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78c8e03700 1 -- 172.21.15.162:0/2917863650 learned_addr learned my addr 172.21.15.162:0/2917863650 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4176093329 0 0) 0x7f78c41a9740 con 0x7f78c4109de0 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78b00035b0 con 0x7f78c4109de0 2024-04-27T01:11:09.962 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3357442647 0 0) 0x7f78c41068b0 con 0x7f78c4101150 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c41a9740 con 0x7f78c4101150 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 926875379 0 0) 0x7f78c41a9740 con 0x7f78c4101150 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78c41068b0 con 0x7f78c4101150 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 461180732 0 0) 0x7f78b00035b0 con 0x7f78c4109de0 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78c41a9740 con 0x7f78c4109de0 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f78c00014d0 con 0x7f78c4101150 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f78b4001910 con 0x7f78c4109de0 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3315695776 0 0) 0x7f78c41a8560 con 0x7f78c410d980 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78b00035b0 con 0x7f78c410d980 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 7966530 0 0) 0x7f78c41068b0 con 0x7f78c4101150 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.969+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 >> v1:172.21.15.162:6790/0 conn(0x7f78c4109de0 legacy=0x7f78c4104d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.970+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 >> v1:172.21.15.181:6789/0 conn(0x7f78c410d980 legacy=0x7f78c41a5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.970+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c41aa920 con 0x7f78c4101150 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.970+0000 7f78cb868700 1 -- 172.21.15.162:0/2917863650 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78c41a75b0 con 0x7f78c4101150 2024-04-27T01:11:09.963 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.970+0000 7f78cb868700 1 -- 172.21.15.162:0/2917863650 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78c41a7a70 con 0x7f78c4101150 2024-04-27T01:11:09.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.970+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f78c0001b30 con 0x7f78c4101150 2024-04-27T01:11:09.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.970+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f78c0004a80 con 0x7f78c4101150 2024-04-27T01:11:09.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.970+0000 7f78cb868700 1 -- 172.21.15.162:0/2917863650 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a8005120 con 0x7f78c4101150 2024-04-27T01:11:09.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.971+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f78c0003960 con 0x7f78c4101150 2024-04-27T01:11:09.964 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.971+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7f78c0088460 con 0x7f78c4101150 2024-04-27T01:11:09.966 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:09.976+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f78c0055b60 con 0x7f78c4101150 2024-04-27T01:11:10.181 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:10.191+0000 7f78cb868700 1 -- 172.21.15.162:0/2917863650 --> v1:172.21.15.162:6800/624703537 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f78a8000b90 con 0x7f78b006ee80 2024-04-27T01:11:10.192 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:10.202+0000 7f78ba7fc700 1 -- 172.21.15.162:0/2917863650 <== mgr.14622 v1:172.21.15.162:6800/624703537 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346507 (unknown 2965378022 0 587624126) 0x7f78a8000b90 con 0x7f78b006ee80 2024-04-27T01:11:10.197 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:11:10.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:10.205+0000 7f78cb868700 1 -- 172.21.15.162:0/2917863650 >> v1:172.21.15.162:6800/624703537 conn(0x7f78b006ee80 legacy=0x7f78b0071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:10.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:10.205+0000 7f78cb868700 1 -- 172.21.15.162:0/2917863650 >> v1:172.21.15.162:6789/0 conn(0x7f78c4101150 legacy=0x7f78c4104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:10.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:10.206+0000 7f78cb868700 1 -- 172.21.15.162:0/2917863650 shutdown_connections 2024-04-27T01:11:10.199 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:10.206+0000 7f78cb868700 1 -- 172.21.15.162:0/2917863650 >> 172.21.15.162:0/2917863650 conn(0x7f78c40fca00 msgr2=0x7f78c406aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:10.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:10.206+0000 7f78cb868700 1 -- 172.21.15.162:0/2917863650 shutdown_connections 2024-04-27T01:11:10.200 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:10.206+0000 7f78cb868700 1 -- 172.21.15.162:0/2917863650 wait complete. 2024-04-27T01:11:10.200 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-04-27T01:11:10.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:10 smithi162 ceph-mon[37680]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:10.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:10 smithi162 ceph-mon[26714]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:11.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:10 smithi181 ceph-mon[30237]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:11.745 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-04-27T01:11:09.082430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1289,"num_read_kb":1103,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":797,"ondisk_log_size":797,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":252724,"kb_used_data":7452,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749708492,"statfs":{"total":767960285184,"available":767701495808,"internally_reserved":0,"allocated":7630848,"data_stored":4046953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003102"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028470+0000","last_change":"2024-04-27T01:09:57.993694+0000","last_active":"2024-04-27T01:10:31.028470+0000","last_peered":"2024-04-27T01:10:31.028470+0000","last_clean":"2024-04-27T01:10:31.028470+0000","last_became_active":"2024-04-27T01:09:57.993370+0000","last_became_peered":"2024-04-27T01:09:57.993370+0000","last_unstale":"2024-04-27T01:10:31.028470+0000","last_undegraded":"2024-04-27T01:10:31.028470+0000","last_fullsized":"2024-04-27T01:10:31.028470+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.224904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.033410+0000","last_change":"2024-04-27T01:09:55.991919+0000","last_active":"2024-04-27T01:10:31.033410+0000","last_peered":"2024-04-27T01:10:31.033410+0000","last_clean":"2024-04-27T01:10:31.033410+0000","last_became_active":"2024-04-27T01:09:55.991263+0000","last_became_peered":"2024-04-27T01:09:55.991263+0000","last_unstale":"2024-04-27T01:10:31.033410+0000","last_undegraded":"2024-04-27T01:10:31.033410+0000","last_fullsized":"2024-04-27T01:10:31.033410+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:55:14.625326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.214033+0000","last_change":"2024-04-27T01:09:51.968754+0000","last_active":"2024-04-27T01:10:31.214033+0000","last_peered":"2024-04-27T01:10:31.214033+0000","last_clean":"2024-04-27T01:10:31.214033+0000","last_became_active":"2024-04-27T01:09:51.968457+0000","last_became_peered":"2024-04-27T01:09:51.968457+0000","last_unstale":"2024-04-27T01:10:31.214033+0000","last_undegraded":"2024-04-27T01:10:31.214033+0000","last_fullsized":"2024-04-27T01:10:31.214033+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:16:11.811934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'16","reported_seq":76,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028454+0000","last_change":"2024-04-27T01:10:22.096093+0000","last_active":"2024-04-27T01:10:31.028454+0000","last_peered":"2024-04-27T01:10:31.028454+0000","last_clean":"2024-04-27T01:10:31.028454+0000","last_became_active":"2024-04-27T01:10:22.082641+0000","last_became_peered":"2024-04-27T01:10:22.082641+0000","last_unstale":"2024-04-27T01:10:31.028454+0000","last_undegraded":"2024-04-27T01:10:31.028454+0000","last_fullsized":"2024-04-27T01:10:31.028454+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:44.047387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202083+0000","last_change":"2024-04-27T01:09:58.493006+0000","last_active":"2024-04-27T01:10:31.202083+0000","last_peered":"2024-04-27T01:10:31.202083+0000","last_clean":"2024-04-27T01:10:31.202083+0000","last_became_active":"2024-04-27T01:09:58.492609+0000","last_became_peered":"2024-04-27T01:09:58.492609+0000","last_unstale":"2024-04-27T01:10:31.202083+0000","last_undegraded":"2024-04-27T01:10:31.202083+0000","last_fullsized":"2024-04-27T01:10:31.202083+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:27.598635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028509+0000","last_change":"2024-04-27T01:09:55.988582+0000","last_active":"2024-04-27T01:10:31.028509+0000","last_peered":"2024-04-27T01:10:31.028509+0000","last_clean":"2024-04-27T01:10:31.028509+0000","last_became_active":"2024-04-27T01:09:55.988296+0000","last_became_peered":"2024-04-27T01:09:55.988296+0000","last_unstale":"2024-04-27T01:10:31.028509+0000","last_undegraded":"2024-04-27T01:10:31.028509+0000","last_fullsized":"2024-04-27T01:10:31.028509+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:54.291915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.203557+0000","last_change":"2024-04-27T01:09:51.977650+0000","last_active":"2024-04-27T01:10:31.203557+0000","last_peered":"2024-04-27T01:10:31.203557+0000","last_clean":"2024-04-27T01:10:31.203557+0000","last_became_active":"2024-04-27T01:09:51.977352+0000","last_became_peered":"2024-04-27T01:09:51.977352+0000","last_unstale":"2024-04-27T01:10:31.203557+0000","last_undegraded":"2024-04-27T01:10:31.203557+0000","last_fullsized":"2024-04-27T01:10:31.203557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:15.672832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'39","reported_seq":102,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.204399+0000","last_change":"2024-04-27T01:09:53.980284+0000","last_active":"2024-04-27T01:10:31.204399+0000","last_peered":"2024-04-27T01:10:31.204399+0000","last_clean":"2024-04-27T01:10:31.204399+0000","last_became_active":"2024-04-27T01:09:53.979869+0000","last_became_peered":"2024-04-27T01:09:53.979869+0000","last_unstale":"2024-04-27T01:10:31.204399+0000","last_undegraded":"2024-04-27T01:10:31.204399+0000","last_fullsized":"2024-04-27T01:10:31.204399+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:56.069731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202477+0000","last_change":"2024-04-27T01:09:58.002523+0000","last_active":"2024-04-27T01:10:31.202477+0000","last_peered":"2024-04-27T01:10:31.202477+0000","last_clean":"2024-04-27T01:10:31.202477+0000","last_became_active":"2024-04-27T01:09:58.002166+0000","last_became_peered":"2024-04-27T01:09:58.002166+0000","last_unstale":"2024-04-27T01:10:31.202477+0000","last_undegraded":"2024-04-27T01:10:31.202477+0000","last_fullsized":"2024-04-27T01:10:31.202477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:39:05.197998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193278+0000","last_change":"2024-04-27T01:09:55.984659+0000","last_active":"2024-04-27T01:10:31.193278+0000","last_peered":"2024-04-27T01:10:31.193278+0000","last_clean":"2024-04-27T01:10:31.193278+0000","last_became_active":"2024-04-27T01:09:55.984163+0000","last_became_peered":"2024-04-27T01:09:55.984163+0000","last_unstale":"2024-04-27T01:10:31.193278+0000","last_undegraded":"2024-04-27T01:10:31.193278+0000","last_fullsized":"2024-04-27T01:10:31.193278+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:47.572049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194142+0000","last_change":"2024-04-27T01:09:51.975375+0000","last_active":"2024-04-27T01:10:31.194142+0000","last_peered":"2024-04-27T01:10:31.194142+0000","last_clean":"2024-04-27T01:10:31.194142+0000","last_became_active":"2024-04-27T01:09:51.975035+0000","last_became_peered":"2024-04-27T01:09:51.975035+0000","last_unstale":"2024-04-27T01:10:31.194142+0000","last_undegraded":"2024-04-27T01:10:31.194142+0000","last_fullsized":"2024-04-27T01:10:31.194142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:34.939844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'23","reported_seq":71,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028627+0000","last_change":"2024-04-27T01:09:53.979601+0000","last_active":"2024-04-27T01:10:31.028627+0000","last_peered":"2024-04-27T01:10:31.028627+0000","last_clean":"2024-04-27T01:10:31.028627+0000","last_became_active":"2024-04-27T01:09:53.979321+0000","last_became_peered":"2024-04-27T01:09:53.979321+0000","last_unstale":"2024-04-27T01:10:31.028627+0000","last_undegraded":"2024-04-27T01:10:31.028627+0000","last_fullsized":"2024-04-27T01:10:31.028627+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:38.136474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193198+0000","last_change":"2024-04-27T01:09:57.998811+0000","last_active":"2024-04-27T01:10:31.193198+0000","last_peered":"2024-04-27T01:10:31.193198+0000","last_clean":"2024-04-27T01:10:31.193198+0000","last_became_active":"2024-04-27T01:09:57.998508+0000","last_became_peered":"2024-04-27T01:09:57.998508+0000","last_unstale":"2024-04-27T01:10:31.193198+0000","last_undegraded":"2024-04-27T01:10:31.193198+0000","last_fullsized":"2024-04-27T01:10:31.193198+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:17.095866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193996+0000","last_change":"2024-04-27T01:09:55.989483+0000","last_active":"2024-04-27T01:10:31.193996+0000","last_peered":"2024-04-27T01:10:31.193996+0000","last_clean":"2024-04-27T01:10:31.193996+0000","last_became_active":"2024-04-27T01:09:55.989212+0000","last_became_peered":"2024-04-27T01:09:55.989212+0000","last_unstale":"2024-04-27T01:10:31.193996+0000","last_undegraded":"2024-04-27T01:10:31.193996+0000","last_fullsized":"2024-04-27T01:10:31.193996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:10.247737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193655+0000","last_change":"2024-04-27T01:09:51.976098+0000","last_active":"2024-04-27T01:10:31.193655+0000","last_peered":"2024-04-27T01:10:31.193655+0000","last_clean":"2024-04-27T01:10:31.193655+0000","last_became_active":"2024-04-27T01:09:51.975334+0000","last_became_peered":"2024-04-27T01:09:51.975334+0000","last_unstale":"2024-04-27T01:10:31.193655+0000","last_undegraded":"2024-04-27T01:10:31.193655+0000","last_fullsized":"2024-04-27T01:10:31.193655+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:19.106542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'11","reported_seq":51,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028901+0000","last_change":"2024-04-27T01:09:53.985313+0000","last_active":"2024-04-27T01:10:31.028901+0000","last_peered":"2024-04-27T01:10:31.028901+0000","last_clean":"2024-04-27T01:10:31.028901+0000","last_became_active":"2024-04-27T01:09:53.984991+0000","last_became_peered":"2024-04-27T01:09:53.984991+0000","last_unstale":"2024-04-27T01:10:31.028901+0000","last_undegraded":"2024-04-27T01:10:31.028901+0000","last_fullsized":"2024-04-27T01:10:31.028901+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:42:01.646439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029284+0000","last_change":"2024-04-27T01:09:58.490814+0000","last_active":"2024-04-27T01:10:31.029284+0000","last_peered":"2024-04-27T01:10:31.029284+0000","last_clean":"2024-04-27T01:10:31.029284+0000","last_became_active":"2024-04-27T01:09:58.490510+0000","last_became_peered":"2024-04-27T01:09:58.490510+0000","last_unstale":"2024-04-27T01:10:31.029284+0000","last_undegraded":"2024-04-27T01:10:31.029284+0000","last_fullsized":"2024-04-27T01:10:31.029284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:47:00.605910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.034852+0000","last_change":"2024-04-27T01:09:55.994105+0000","last_active":"2024-04-27T01:10:31.034852+0000","last_peered":"2024-04-27T01:10:31.034852+0000","last_clean":"2024-04-27T01:10:31.034852+0000","last_became_active":"2024-04-27T01:09:55.993581+0000","last_became_peered":"2024-04-27T01:09:55.993581+0000","last_unstale":"2024-04-27T01:10:31.034852+0000","last_undegraded":"2024-04-27T01:10:31.034852+0000","last_fullsized":"2024-04-27T01:10:31.034852+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:57:12.005476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.208665+0000","last_change":"2024-04-27T01:09:51.976414+0000","last_active":"2024-04-27T01:10:31.208665+0000","last_peered":"2024-04-27T01:10:31.208665+0000","last_clean":"2024-04-27T01:10:31.208665+0000","last_became_active":"2024-04-27T01:09:51.976154+0000","last_became_peered":"2024-04-27T01:09:51.976154+0000","last_unstale":"2024-04-27T01:10:31.208665+0000","last_undegraded":"2024-04-27T01:10:31.208665+0000","last_fullsized":"2024-04-27T01:10:31.208665+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:20.334383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.030298+0000","last_change":"2024-04-27T01:09:53.980941+0000","last_active":"2024-04-27T01:10:31.030298+0000","last_peered":"2024-04-27T01:10:31.030298+0000","last_clean":"2024-04-27T01:10:31.030298+0000","last_became_active":"2024-04-27T01:09:53.980600+0000","last_became_peered":"2024-04-27T01:09:53.980600+0000","last_unstale":"2024-04-27T01:10:31.030298+0000","last_undegraded":"2024-04-27T01:10:31.030298+0000","last_fullsized":"2024-04-27T01:10:31.030298+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:38:47.989649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.203924+0000","last_change":"2024-04-27T01:09:58.492176+0000","last_active":"2024-04-27T01:10:31.203924+0000","last_peered":"2024-04-27T01:10:31.203924+0000","last_clean":"2024-04-27T01:10:31.203924+0000","last_became_active":"2024-04-27T01:09:58.491717+0000","last_became_peered":"2024-04-27T01:09:58.491717+0000","last_unstale":"2024-04-27T01:10:31.203924+0000","last_undegraded":"2024-04-27T01:10:31.203924+0000","last_fullsized":"2024-04-27T01:10:31.203924+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:06:09.949561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028765+0000","last_change":"2024-04-27T01:09:55.992349+0000","last_active":"2024-04-27T01:10:31.028765+0000","last_peered":"2024-04-27T01:10:31.028765+0000","last_clean":"2024-04-27T01:10:31.028765+0000","last_became_active":"2024-04-27T01:09:55.992068+0000","last_became_peered":"2024-04-27T01:09:55.992068+0000","last_unstale":"2024-04-27T01:10:31.028765+0000","last_undegraded":"2024-04-27T01:10:31.028765+0000","last_fullsized":"2024-04-27T01:10:31.028765+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:15.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028648+0000","last_change":"2024-04-27T01:09:51.978740+0000","last_active":"2024-04-27T01:10:31.028648+0000","last_peered":"2024-04-27T01:10:31.028648+0000","last_clean":"2024-04-27T01:10:31.028648+0000","last_became_active":"2024-04-27T01:09:51.978419+0000","last_became_peered":"2024-04-27T01:09:51.978419+0000","last_unstale":"2024-04-27T01:10:31.028648+0000","last_undegraded":"2024-04-27T01:10:31.028648+0000","last_fullsized":"2024-04-27T01:10:31.028648+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:23:51.854276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'18","reported_seq":64,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.201801+0000","last_change":"2024-04-27T01:09:53.985482+0000","last_active":"2024-04-27T01:10:31.201801+0000","last_peered":"2024-04-27T01:10:31.201801+0000","last_clean":"2024-04-27T01:10:31.201801+0000","last_became_active":"2024-04-27T01:09:53.985099+0000","last_became_peered":"2024-04-27T01:09:53.985099+0000","last_unstale":"2024-04-27T01:10:31.201801+0000","last_undegraded":"2024-04-27T01:10:31.201801+0000","last_fullsized":"2024-04-27T01:10:31.201801+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:31:13.943321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195037+0000","last_change":"2024-04-27T01:09:57.996723+0000","last_active":"2024-04-27T01:10:31.195037+0000","last_peered":"2024-04-27T01:10:31.195037+0000","last_clean":"2024-04-27T01:10:31.195037+0000","last_became_active":"2024-04-27T01:09:57.995767+0000","last_became_peered":"2024-04-27T01:09:57.995767+0000","last_unstale":"2024-04-27T01:10:31.195037+0000","last_undegraded":"2024-04-27T01:10:31.195037+0000","last_fullsized":"2024-04-27T01:10:31.195037+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:00.665929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"69'8","reported_seq":52,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040505+0000","last_change":"2024-04-27T01:09:55.987377+0000","last_active":"2024-04-27T01:11:02.040505+0000","last_peered":"2024-04-27T01:11:02.040505+0000","last_clean":"2024-04-27T01:11:02.040505+0000","last_became_active":"2024-04-27T01:09:55.987091+0000","last_became_peered":"2024-04-27T01:09:55.987091+0000","last_unstale":"2024-04-27T01:11:02.040505+0000","last_undegraded":"2024-04-27T01:11:02.040505+0000","last_fullsized":"2024-04-27T01:11:02.040505+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:10:08.810589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.033500+0000","last_change":"2024-04-27T01:09:51.975349+0000","last_active":"2024-04-27T01:10:31.033500+0000","last_peered":"2024-04-27T01:10:31.033500+0000","last_clean":"2024-04-27T01:10:31.033500+0000","last_became_active":"2024-04-27T01:09:51.975076+0000","last_became_peered":"2024-04-27T01:09:51.975076+0000","last_unstale":"2024-04-27T01:10:31.033500+0000","last_undegraded":"2024-04-27T01:10:31.033500+0000","last_fullsized":"2024-04-27T01:10:31.033500+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:52.747264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'16","reported_seq":56,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.212917+0000","last_change":"2024-04-27T01:09:54.097378+0000","last_active":"2024-04-27T01:10:31.212917+0000","last_peered":"2024-04-27T01:10:31.212917+0000","last_clean":"2024-04-27T01:10:31.212917+0000","last_became_active":"2024-04-27T01:09:54.097109+0000","last_became_peered":"2024-04-27T01:09:54.097109+0000","last_unstale":"2024-04-27T01:10:31.212917+0000","last_undegraded":"2024-04-27T01:10:31.212917+0000","last_fullsized":"2024-04-27T01:10:31.212917+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:29:15.858403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.034995+0000","last_change":"2024-04-27T01:09:57.995773+0000","last_active":"2024-04-27T01:10:31.034995+0000","last_peered":"2024-04-27T01:10:31.034995+0000","last_clean":"2024-04-27T01:10:31.034995+0000","last_became_active":"2024-04-27T01:09:57.995241+0000","last_became_peered":"2024-04-27T01:09:57.995241+0000","last_unstale":"2024-04-27T01:10:31.034995+0000","last_undegraded":"2024-04-27T01:10:31.034995+0000","last_fullsized":"2024-04-27T01:10:31.034995+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:54:52.336231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.213294+0000","last_change":"2024-04-27T01:09:55.983435+0000","last_active":"2024-04-27T01:10:31.213294+0000","last_peered":"2024-04-27T01:10:31.213294+0000","last_clean":"2024-04-27T01:10:31.213294+0000","last_became_active":"2024-04-27T01:09:55.983067+0000","last_became_peered":"2024-04-27T01:09:55.983067+0000","last_unstale":"2024-04-27T01:10:31.213294+0000","last_undegraded":"2024-04-27T01:10:31.213294+0000","last_fullsized":"2024-04-27T01:10:31.213294+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:49.791414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200589+0000","last_change":"2024-04-27T01:09:51.977196+0000","last_active":"2024-04-27T01:10:31.200589+0000","last_peered":"2024-04-27T01:10:31.200589+0000","last_clean":"2024-04-27T01:10:31.200589+0000","last_became_active":"2024-04-27T01:09:51.976906+0000","last_became_peered":"2024-04-27T01:09:51.976906+0000","last_unstale":"2024-04-27T01:10:31.200589+0000","last_undegraded":"2024-04-27T01:10:31.200589+0000","last_fullsized":"2024-04-27T01:10:31.200589+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:19:42.666108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":60,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195406+0000","last_change":"2024-04-27T01:09:53.982622+0000","last_active":"2024-04-27T01:10:31.195406+0000","last_peered":"2024-04-27T01:10:31.195406+0000","last_clean":"2024-04-27T01:10:31.195406+0000","last_became_active":"2024-04-27T01:09:53.982311+0000","last_became_peered":"2024-04-27T01:09:53.982311+0000","last_unstale":"2024-04-27T01:10:31.195406+0000","last_undegraded":"2024-04-27T01:10:31.195406+0000","last_fullsized":"2024-04-27T01:10:31.195406+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:05.219310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193899+0000","last_change":"2024-04-27T01:09:57.995625+0000","last_active":"2024-04-27T01:10:31.193899+0000","last_peered":"2024-04-27T01:10:31.193899+0000","last_clean":"2024-04-27T01:10:31.193899+0000","last_became_active":"2024-04-27T01:09:57.995033+0000","last_became_peered":"2024-04-27T01:09:57.995033+0000","last_unstale":"2024-04-27T01:10:31.193899+0000","last_undegraded":"2024-04-27T01:10:31.193899+0000","last_fullsized":"2024-04-27T01:10:31.193899+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:40:52.027696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193194+0000","last_change":"2024-04-27T01:09:55.986992+0000","last_active":"2024-04-27T01:10:31.193194+0000","last_peered":"2024-04-27T01:10:31.193194+0000","last_clean":"2024-04-27T01:10:31.193194+0000","last_became_active":"2024-04-27T01:09:55.986653+0000","last_became_peered":"2024-04-27T01:09:55.986653+0000","last_unstale":"2024-04-27T01:10:31.193194+0000","last_undegraded":"2024-04-27T01:10:31.193194+0000","last_fullsized":"2024-04-27T01:10:31.193194+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:48.395497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.214125+0000","last_change":"2024-04-27T01:09:51.980028+0000","last_active":"2024-04-27T01:10:31.214125+0000","last_peered":"2024-04-27T01:10:31.214125+0000","last_clean":"2024-04-27T01:10:31.214125+0000","last_became_active":"2024-04-27T01:09:51.979725+0000","last_became_peered":"2024-04-27T01:09:51.979725+0000","last_unstale":"2024-04-27T01:10:31.214125+0000","last_undegraded":"2024-04-27T01:10:31.214125+0000","last_fullsized":"2024-04-27T01:10:31.214125+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:16.016040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":35,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200685+0000","last_change":"2024-04-27T01:09:53.983929+0000","last_active":"2024-04-27T01:10:31.200685+0000","last_peered":"2024-04-27T01:10:31.200685+0000","last_clean":"2024-04-27T01:10:31.200685+0000","last_became_active":"2024-04-27T01:09:53.983592+0000","last_became_peered":"2024-04-27T01:09:53.983592+0000","last_unstale":"2024-04-27T01:10:31.200685+0000","last_undegraded":"2024-04-27T01:10:31.200685+0000","last_fullsized":"2024-04-27T01:10:31.200685+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:53.243435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200297+0000","last_change":"2024-04-27T01:09:58.492509+0000","last_active":"2024-04-27T01:10:31.200297+0000","last_peered":"2024-04-27T01:10:31.200297+0000","last_clean":"2024-04-27T01:10:31.200297+0000","last_became_active":"2024-04-27T01:09:58.492189+0000","last_became_peered":"2024-04-27T01:09:58.492189+0000","last_unstale":"2024-04-27T01:10:31.200297+0000","last_undegraded":"2024-04-27T01:10:31.200297+0000","last_fullsized":"2024-04-27T01:10:31.200297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:41:44.975123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194815+0000","last_change":"2024-04-27T01:09:55.991580+0000","last_active":"2024-04-27T01:10:31.194815+0000","last_peered":"2024-04-27T01:10:31.194815+0000","last_clean":"2024-04-27T01:10:31.194815+0000","last_became_active":"2024-04-27T01:09:55.991287+0000","last_became_peered":"2024-04-27T01:09:55.991287+0000","last_unstale":"2024-04-27T01:10:31.194815+0000","last_undegraded":"2024-04-27T01:10:31.194815+0000","last_fullsized":"2024-04-27T01:10:31.194815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:42:08.400885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195058+0000","last_change":"2024-04-27T01:09:51.976544+0000","last_active":"2024-04-27T01:10:31.195058+0000","last_peered":"2024-04-27T01:10:31.195058+0000","last_clean":"2024-04-27T01:10:31.195058+0000","last_became_active":"2024-04-27T01:09:51.976196+0000","last_became_peered":"2024-04-27T01:09:51.976196+0000","last_unstale":"2024-04-27T01:10:31.195058+0000","last_undegraded":"2024-04-27T01:10:31.195058+0000","last_fullsized":"2024-04-27T01:10:31.195058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:53:45.288962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"69'17","reported_seq":60,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.204263+0000","last_change":"2024-04-27T01:09:54.096077+0000","last_active":"2024-04-27T01:10:31.204263+0000","last_peered":"2024-04-27T01:10:31.204263+0000","last_clean":"2024-04-27T01:10:31.204263+0000","last_became_active":"2024-04-27T01:09:54.095717+0000","last_became_peered":"2024-04-27T01:09:54.095717+0000","last_unstale":"2024-04-27T01:10:31.204263+0000","last_undegraded":"2024-04-27T01:10:31.204263+0000","last_fullsized":"2024-04-27T01:10:31.204263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:13:36.573405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202175+0000","last_change":"2024-04-27T01:09:58.003824+0000","last_active":"2024-04-27T01:10:31.202175+0000","last_peered":"2024-04-27T01:10:31.202175+0000","last_clean":"2024-04-27T01:10:31.202175+0000","last_became_active":"2024-04-27T01:09:58.003356+0000","last_became_peered":"2024-04-27T01:09:58.003356+0000","last_unstale":"2024-04-27T01:10:31.202175+0000","last_undegraded":"2024-04-27T01:10:31.202175+0000","last_fullsized":"2024-04-27T01:10:31.202175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:04.144673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202185+0000","last_change":"2024-04-27T01:09:55.995358+0000","last_active":"2024-04-27T01:10:31.202185+0000","last_peered":"2024-04-27T01:10:31.202185+0000","last_clean":"2024-04-27T01:10:31.202185+0000","last_became_active":"2024-04-27T01:09:55.994787+0000","last_became_peered":"2024-04-27T01:09:55.994787+0000","last_unstale":"2024-04-27T01:10:31.202185+0000","last_undegraded":"2024-04-27T01:10:31.202185+0000","last_fullsized":"2024-04-27T01:10:31.202185+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:43:08.388998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194645+0000","last_change":"2024-04-27T01:09:51.981304+0000","last_active":"2024-04-27T01:10:31.194645+0000","last_peered":"2024-04-27T01:10:31.194645+0000","last_clean":"2024-04-27T01:10:31.194645+0000","last_became_active":"2024-04-27T01:09:51.980959+0000","last_became_peered":"2024-04-27T01:09:51.980959+0000","last_unstale":"2024-04-27T01:10:31.194645+0000","last_undegraded":"2024-04-27T01:10:31.194645+0000","last_fullsized":"2024-04-27T01:10:31.194645+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:34:55.918660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.034264+0000","last_change":"2024-04-27T01:09:53.984436+0000","last_active":"2024-04-27T01:10:31.034264+0000","last_peered":"2024-04-27T01:10:31.034264+0000","last_clean":"2024-04-27T01:10:31.034264+0000","last_became_active":"2024-04-27T01:09:53.984121+0000","last_became_peered":"2024-04-27T01:09:53.984121+0000","last_unstale":"2024-04-27T01:10:31.034264+0000","last_undegraded":"2024-04-27T01:10:31.034264+0000","last_fullsized":"2024-04-27T01:10:31.034264+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:25.560995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.204159+0000","last_change":"2024-04-27T01:09:58.000375+0000","last_active":"2024-04-27T01:10:31.204159+0000","last_peered":"2024-04-27T01:10:31.204159+0000","last_clean":"2024-04-27T01:10:31.204159+0000","last_became_active":"2024-04-27T01:09:57.999943+0000","last_became_peered":"2024-04-27T01:09:57.999943+0000","last_unstale":"2024-04-27T01:10:31.204159+0000","last_undegraded":"2024-04-27T01:10:31.204159+0000","last_fullsized":"2024-04-27T01:10:31.204159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:03.658855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.034036+0000","last_change":"2024-04-27T01:09:55.993241+0000","last_active":"2024-04-27T01:10:31.034036+0000","last_peered":"2024-04-27T01:10:31.034036+0000","last_clean":"2024-04-27T01:10:31.034036+0000","last_became_active":"2024-04-27T01:09:55.992901+0000","last_became_peered":"2024-04-27T01:09:55.992901+0000","last_unstale":"2024-04-27T01:10:31.034036+0000","last_undegraded":"2024-04-27T01:10:31.034036+0000","last_fullsized":"2024-04-27T01:10:31.034036+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:21:15.285498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028964+0000","last_change":"2024-04-27T01:09:51.979581+0000","last_active":"2024-04-27T01:10:31.028964+0000","last_peered":"2024-04-27T01:10:31.028964+0000","last_clean":"2024-04-27T01:10:31.028964+0000","last_became_active":"2024-04-27T01:09:51.979251+0000","last_became_peered":"2024-04-27T01:09:51.979251+0000","last_unstale":"2024-04-27T01:10:31.028964+0000","last_undegraded":"2024-04-27T01:10:31.028964+0000","last_fullsized":"2024-04-27T01:10:31.028964+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:04:15.479829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":66,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029226+0000","last_change":"2024-04-27T01:10:22.102174+0000","last_active":"2024-04-27T01:10:31.029226+0000","last_peered":"2024-04-27T01:10:31.029226+0000","last_clean":"2024-04-27T01:10:31.029226+0000","last_became_active":"2024-04-27T01:10:22.084998+0000","last_became_peered":"2024-04-27T01:10:22.084998+0000","last_unstale":"2024-04-27T01:10:31.029226+0000","last_undegraded":"2024-04-27T01:10:31.029226+0000","last_fullsized":"2024-04-27T01:10:31.029226+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:31.738300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.207946+0000","last_change":"2024-04-27T01:09:58.003556+0000","last_active":"2024-04-27T01:10:31.207946+0000","last_peered":"2024-04-27T01:10:31.207946+0000","last_clean":"2024-04-27T01:10:31.207946+0000","last_became_active":"2024-04-27T01:09:58.003224+0000","last_became_peered":"2024-04-27T01:09:58.003224+0000","last_unstale":"2024-04-27T01:10:31.207946+0000","last_undegraded":"2024-04-27T01:10:31.207946+0000","last_fullsized":"2024-04-27T01:10:31.207946+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:46:55.737761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":49,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040532+0000","last_change":"2024-04-27T01:09:55.990672+0000","last_active":"2024-04-27T01:11:02.040532+0000","last_peered":"2024-04-27T01:11:02.040532+0000","last_clean":"2024-04-27T01:11:02.040532+0000","last_became_active":"2024-04-27T01:09:55.989696+0000","last_became_peered":"2024-04-27T01:09:55.989696+0000","last_unstale":"2024-04-27T01:11:02.040532+0000","last_undegraded":"2024-04-27T01:11:02.040532+0000","last_fullsized":"2024-04-27T01:11:02.040532+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:09.173488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193344+0000","last_change":"2024-04-27T01:09:51.978172+0000","last_active":"2024-04-27T01:10:31.193344+0000","last_peered":"2024-04-27T01:10:31.193344+0000","last_clean":"2024-04-27T01:10:31.193344+0000","last_became_active":"2024-04-27T01:09:51.977820+0000","last_became_peered":"2024-04-27T01:09:51.977820+0000","last_unstale":"2024-04-27T01:10:31.193344+0000","last_undegraded":"2024-04-27T01:10:31.193344+0000","last_fullsized":"2024-04-27T01:10:31.193344+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:58.795962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'23","reported_seq":69,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.201070+0000","last_change":"2024-04-27T01:09:54.096114+0000","last_active":"2024-04-27T01:10:31.201070+0000","last_peered":"2024-04-27T01:10:31.201070+0000","last_clean":"2024-04-27T01:10:31.201070+0000","last_became_active":"2024-04-27T01:09:54.095824+0000","last_became_peered":"2024-04-27T01:09:54.095824+0000","last_unstale":"2024-04-27T01:10:31.201070+0000","last_undegraded":"2024-04-27T01:10:31.201070+0000","last_fullsized":"2024-04-27T01:10:31.201070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:37.335086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029748+0000","last_change":"2024-04-27T01:09:57.996025+0000","last_active":"2024-04-27T01:10:31.029748+0000","last_peered":"2024-04-27T01:10:31.029748+0000","last_clean":"2024-04-27T01:10:31.029748+0000","last_became_active":"2024-04-27T01:09:57.995746+0000","last_became_peered":"2024-04-27T01:09:57.995746+0000","last_unstale":"2024-04-27T01:10:31.029748+0000","last_undegraded":"2024-04-27T01:10:31.029748+0000","last_fullsized":"2024-04-27T01:10:31.029748+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:23:54.706381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":49,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040562+0000","last_change":"2024-04-27T01:09:55.990395+0000","last_active":"2024-04-27T01:11:02.040562+0000","last_peered":"2024-04-27T01:11:02.040562+0000","last_clean":"2024-04-27T01:11:02.040562+0000","last_became_active":"2024-04-27T01:09:55.990086+0000","last_became_peered":"2024-04-27T01:09:55.990086+0000","last_unstale":"2024-04-27T01:11:02.040562+0000","last_undegraded":"2024-04-27T01:11:02.040562+0000","last_fullsized":"2024-04-27T01:11:02.040562+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:03.299934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.209784+0000","last_change":"2024-04-27T01:09:51.976764+0000","last_active":"2024-04-27T01:10:31.209784+0000","last_peered":"2024-04-27T01:10:31.209784+0000","last_clean":"2024-04-27T01:10:31.209784+0000","last_became_active":"2024-04-27T01:09:51.976161+0000","last_became_peered":"2024-04-27T01:09:51.976161+0000","last_unstale":"2024-04-27T01:10:31.209784+0000","last_undegraded":"2024-04-27T01:10:31.209784+0000","last_fullsized":"2024-04-27T01:10:31.209784+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:06:04.577915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'15","reported_seq":57,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194507+0000","last_change":"2024-04-27T01:09:54.097924+0000","last_active":"2024-04-27T01:10:31.194507+0000","last_peered":"2024-04-27T01:10:31.194507+0000","last_clean":"2024-04-27T01:10:31.194507+0000","last_became_active":"2024-04-27T01:09:54.097583+0000","last_became_peered":"2024-04-27T01:09:54.097583+0000","last_unstale":"2024-04-27T01:10:31.194507+0000","last_undegraded":"2024-04-27T01:10:31.194507+0000","last_fullsized":"2024-04-27T01:10:31.194507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:12:03.954912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.030690+0000","last_change":"2024-04-27T01:09:57.993895+0000","last_active":"2024-04-27T01:10:31.030690+0000","last_peered":"2024-04-27T01:10:31.030690+0000","last_clean":"2024-04-27T01:10:31.030690+0000","last_became_active":"2024-04-27T01:09:57.993564+0000","last_became_peered":"2024-04-27T01:09:57.993564+0000","last_unstale":"2024-04-27T01:10:31.030690+0000","last_undegraded":"2024-04-27T01:10:31.030690+0000","last_fullsized":"2024-04-27T01:10:31.030690+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:12:41.054063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.203834+0000","last_change":"2024-04-27T01:09:55.991784+0000","last_active":"2024-04-27T01:10:31.203834+0000","last_peered":"2024-04-27T01:10:31.203834+0000","last_clean":"2024-04-27T01:10:31.203834+0000","last_became_active":"2024-04-27T01:09:55.991441+0000","last_became_peered":"2024-04-27T01:09:55.991441+0000","last_unstale":"2024-04-27T01:10:31.203834+0000","last_undegraded":"2024-04-27T01:10:31.203834+0000","last_fullsized":"2024-04-27T01:10:31.203834+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:34.613206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193549+0000","last_change":"2024-04-27T01:09:51.977480+0000","last_active":"2024-04-27T01:10:31.193549+0000","last_peered":"2024-04-27T01:10:31.193549+0000","last_clean":"2024-04-27T01:10:31.193549+0000","last_became_active":"2024-04-27T01:09:51.977157+0000","last_became_peered":"2024-04-27T01:09:51.977157+0000","last_unstale":"2024-04-27T01:10:31.193549+0000","last_undegraded":"2024-04-27T01:10:31.193549+0000","last_fullsized":"2024-04-27T01:10:31.193549+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:54.830982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'21","reported_seq":69,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.213088+0000","last_change":"2024-04-27T01:09:54.097131+0000","last_active":"2024-04-27T01:10:31.213088+0000","last_peered":"2024-04-27T01:10:31.213088+0000","last_clean":"2024-04-27T01:10:31.213088+0000","last_became_active":"2024-04-27T01:09:54.096854+0000","last_became_peered":"2024-04-27T01:09:54.096854+0000","last_unstale":"2024-04-27T01:10:31.213088+0000","last_undegraded":"2024-04-27T01:10:31.213088+0000","last_fullsized":"2024-04-27T01:10:31.213088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:45:27.541174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194550+0000","last_change":"2024-04-27T01:09:58.492699+0000","last_active":"2024-04-27T01:10:31.194550+0000","last_peered":"2024-04-27T01:10:31.194550+0000","last_clean":"2024-04-27T01:10:31.194550+0000","last_became_active":"2024-04-27T01:09:58.492357+0000","last_became_peered":"2024-04-27T01:09:58.492357+0000","last_unstale":"2024-04-27T01:10:31.194550+0000","last_undegraded":"2024-04-27T01:10:31.194550+0000","last_fullsized":"2024-04-27T01:10:31.194550+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:17:39.800109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195146+0000","last_change":"2024-04-27T01:09:55.990121+0000","last_active":"2024-04-27T01:10:31.195146+0000","last_peered":"2024-04-27T01:10:31.195146+0000","last_clean":"2024-04-27T01:10:31.195146+0000","last_became_active":"2024-04-27T01:09:55.989627+0000","last_became_peered":"2024-04-27T01:09:55.989627+0000","last_unstale":"2024-04-27T01:10:31.195146+0000","last_undegraded":"2024-04-27T01:10:31.195146+0000","last_fullsized":"2024-04-27T01:10:31.195146+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:37:17.381011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193879+0000","last_change":"2024-04-27T01:09:51.975778+0000","last_active":"2024-04-27T01:10:31.193879+0000","last_peered":"2024-04-27T01:10:31.193879+0000","last_clean":"2024-04-27T01:10:31.193879+0000","last_became_active":"2024-04-27T01:09:51.975469+0000","last_became_peered":"2024-04-27T01:09:51.975469+0000","last_unstale":"2024-04-27T01:10:31.193879+0000","last_undegraded":"2024-04-27T01:10:31.193879+0000","last_fullsized":"2024-04-27T01:10:31.193879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:46:26.134298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'12","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.204024+0000","last_change":"2024-04-27T01:09:54.096271+0000","last_active":"2024-04-27T01:10:31.204024+0000","last_peered":"2024-04-27T01:10:31.204024+0000","last_clean":"2024-04-27T01:10:31.204024+0000","last_became_active":"2024-04-27T01:09:54.095897+0000","last_became_peered":"2024-04-27T01:09:54.095897+0000","last_unstale":"2024-04-27T01:10:31.204024+0000","last_undegraded":"2024-04-27T01:10:31.204024+0000","last_fullsized":"2024-04-27T01:10:31.204024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:54.537542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193843+0000","last_change":"2024-04-27T01:09:58.492112+0000","last_active":"2024-04-27T01:10:31.193843+0000","last_peered":"2024-04-27T01:10:31.193843+0000","last_clean":"2024-04-27T01:10:31.193843+0000","last_became_active":"2024-04-27T01:09:58.491814+0000","last_became_peered":"2024-04-27T01:09:58.491814+0000","last_unstale":"2024-04-27T01:10:31.193843+0000","last_undegraded":"2024-04-27T01:10:31.193843+0000","last_fullsized":"2024-04-27T01:10:31.193843+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:12.127109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195152+0000","last_change":"2024-04-27T01:09:51.984095+0000","last_active":"2024-04-27T01:10:31.195152+0000","last_peered":"2024-04-27T01:10:31.195152+0000","last_clean":"2024-04-27T01:10:31.195152+0000","last_became_active":"2024-04-27T01:09:51.983527+0000","last_became_peered":"2024-04-27T01:09:51.983527+0000","last_unstale":"2024-04-27T01:10:31.195152+0000","last_undegraded":"2024-04-27T01:10:31.195152+0000","last_fullsized":"2024-04-27T01:10:31.195152+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:07.781337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"69'28","reported_seq":79,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193074+0000","last_change":"2024-04-27T01:09:54.098741+0000","last_active":"2024-04-27T01:10:31.193074+0000","last_peered":"2024-04-27T01:10:31.193074+0000","last_clean":"2024-04-27T01:10:31.193074+0000","last_became_active":"2024-04-27T01:09:54.098422+0000","last_became_peered":"2024-04-27T01:09:54.098422+0000","last_unstale":"2024-04-27T01:10:31.193074+0000","last_undegraded":"2024-04-27T01:10:31.193074+0000","last_fullsized":"2024-04-27T01:10:31.193074+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:27.798531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":49,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040520+0000","last_change":"2024-04-27T01:09:55.992444+0000","last_active":"2024-04-27T01:11:02.040520+0000","last_peered":"2024-04-27T01:11:02.040520+0000","last_clean":"2024-04-27T01:11:02.040520+0000","last_became_active":"2024-04-27T01:09:55.991936+0000","last_became_peered":"2024-04-27T01:09:55.991936+0000","last_unstale":"2024-04-27T01:11:02.040520+0000","last_undegraded":"2024-04-27T01:11:02.040520+0000","last_fullsized":"2024-04-27T01:11:02.040520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:22:04.321113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200404+0000","last_change":"2024-04-27T01:09:58.000147+0000","last_active":"2024-04-27T01:10:31.200404+0000","last_peered":"2024-04-27T01:10:31.200404+0000","last_clean":"2024-04-27T01:10:31.200404+0000","last_became_active":"2024-04-27T01:09:57.999476+0000","last_became_peered":"2024-04-27T01:09:57.999476+0000","last_unstale":"2024-04-27T01:10:31.200404+0000","last_undegraded":"2024-04-27T01:10:31.200404+0000","last_fullsized":"2024-04-27T01:10:31.200404+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:07.164832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194030+0000","last_change":"2024-04-27T01:09:51.981342+0000","last_active":"2024-04-27T01:10:31.194030+0000","last_peered":"2024-04-27T01:10:31.194030+0000","last_clean":"2024-04-27T01:10:31.194030+0000","last_became_active":"2024-04-27T01:09:51.981012+0000","last_became_peered":"2024-04-27T01:09:51.981012+0000","last_unstale":"2024-04-27T01:10:31.194030+0000","last_undegraded":"2024-04-27T01:10:31.194030+0000","last_fullsized":"2024-04-27T01:10:31.194030+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:39.300351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'24","reported_seq":76,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029374+0000","last_change":"2024-04-27T01:09:53.980354+0000","last_active":"2024-04-27T01:10:31.029374+0000","last_peered":"2024-04-27T01:10:31.029374+0000","last_clean":"2024-04-27T01:10:31.029374+0000","last_became_active":"2024-04-27T01:09:53.979993+0000","last_became_peered":"2024-04-27T01:09:53.979993+0000","last_unstale":"2024-04-27T01:10:31.029374+0000","last_undegraded":"2024-04-27T01:10:31.029374+0000","last_fullsized":"2024-04-27T01:10:31.029374+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:47.872365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029326+0000","last_change":"2024-04-27T01:09:55.987622+0000","last_active":"2024-04-27T01:10:31.029326+0000","last_peered":"2024-04-27T01:10:31.029326+0000","last_clean":"2024-04-27T01:10:31.029326+0000","last_became_active":"2024-04-27T01:09:55.987286+0000","last_became_peered":"2024-04-27T01:09:55.987286+0000","last_unstale":"2024-04-27T01:10:31.029326+0000","last_undegraded":"2024-04-27T01:10:31.029326+0000","last_fullsized":"2024-04-27T01:10:31.029326+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:59:58.984779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"69'1","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194293+0000","last_change":"2024-04-27T01:09:57.997122+0000","last_active":"2024-04-27T01:10:31.194293+0000","last_peered":"2024-04-27T01:10:31.194293+0000","last_clean":"2024-04-27T01:10:31.194293+0000","last_became_active":"2024-04-27T01:09:57.996499+0000","last_became_peered":"2024-04-27T01:09:57.996499+0000","last_unstale":"2024-04-27T01:10:31.194293+0000","last_undegraded":"2024-04-27T01:10:31.194293+0000","last_fullsized":"2024-04-27T01:10:31.194293+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:50:31.513681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.030144+0000","last_change":"2024-04-27T01:09:55.988725+0000","last_active":"2024-04-27T01:10:31.030144+0000","last_peered":"2024-04-27T01:10:31.030144+0000","last_clean":"2024-04-27T01:10:31.030144+0000","last_became_active":"2024-04-27T01:09:55.988394+0000","last_became_peered":"2024-04-27T01:09:55.988394+0000","last_unstale":"2024-04-27T01:10:31.030144+0000","last_undegraded":"2024-04-27T01:10:31.030144+0000","last_fullsized":"2024-04-27T01:10:31.030144+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:17:19.998896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194879+0000","last_change":"2024-04-27T01:09:51.979406+0000","last_active":"2024-04-27T01:10:31.194879+0000","last_peered":"2024-04-27T01:10:31.194879+0000","last_clean":"2024-04-27T01:10:31.194879+0000","last_became_active":"2024-04-27T01:09:51.979078+0000","last_became_peered":"2024-04-27T01:09:51.979078+0000","last_unstale":"2024-04-27T01:10:31.194879+0000","last_undegraded":"2024-04-27T01:10:31.194879+0000","last_fullsized":"2024-04-27T01:10:31.194879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:06:18.209863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'25","reported_seq":72,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194402+0000","last_change":"2024-04-27T01:09:54.096387+0000","last_active":"2024-04-27T01:10:31.194402+0000","last_peered":"2024-04-27T01:10:31.194402+0000","last_clean":"2024-04-27T01:10:31.194402+0000","last_became_active":"2024-04-27T01:09:54.096056+0000","last_became_peered":"2024-04-27T01:09:54.096056+0000","last_unstale":"2024-04-27T01:10:31.194402+0000","last_undegraded":"2024-04-27T01:10:31.194402+0000","last_fullsized":"2024-04-27T01:10:31.194402+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:17.119595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202303+0000","last_change":"2024-04-27T01:09:58.000860+0000","last_active":"2024-04-27T01:10:31.202303+0000","last_peered":"2024-04-27T01:10:31.202303+0000","last_clean":"2024-04-27T01:10:31.202303+0000","last_became_active":"2024-04-27T01:09:58.000552+0000","last_became_peered":"2024-04-27T01:09:58.000552+0000","last_unstale":"2024-04-27T01:10:31.202303+0000","last_undegraded":"2024-04-27T01:10:31.202303+0000","last_fullsized":"2024-04-27T01:10:31.202303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:38.802150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029932+0000","last_change":"2024-04-27T01:09:55.986655+0000","last_active":"2024-04-27T01:10:31.029932+0000","last_peered":"2024-04-27T01:10:31.029932+0000","last_clean":"2024-04-27T01:10:31.029932+0000","last_became_active":"2024-04-27T01:09:55.986294+0000","last_became_peered":"2024-04-27T01:09:55.986294+0000","last_unstale":"2024-04-27T01:10:31.029932+0000","last_undegraded":"2024-04-27T01:10:31.029932+0000","last_fullsized":"2024-04-27T01:10:31.029932+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:07:26.356079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193752+0000","last_change":"2024-04-27T01:09:51.974394+0000","last_active":"2024-04-27T01:10:31.193752+0000","last_peered":"2024-04-27T01:10:31.193752+0000","last_clean":"2024-04-27T01:10:31.193752+0000","last_became_active":"2024-04-27T01:09:51.974100+0000","last_became_peered":"2024-04-27T01:09:51.974100+0000","last_unstale":"2024-04-27T01:10:31.193752+0000","last_undegraded":"2024-04-27T01:10:31.193752+0000","last_fullsized":"2024-04-27T01:10:31.193752+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:26.036970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"69'15","reported_seq":57,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.208967+0000","last_change":"2024-04-27T01:09:53.985351+0000","last_active":"2024-04-27T01:10:31.208967+0000","last_peered":"2024-04-27T01:10:31.208967+0000","last_clean":"2024-04-27T01:10:31.208967+0000","last_became_active":"2024-04-27T01:09:53.985000+0000","last_became_peered":"2024-04-27T01:09:53.985000+0000","last_unstale":"2024-04-27T01:10:31.208967+0000","last_undegraded":"2024-04-27T01:10:31.208967+0000","last_fullsized":"2024-04-27T01:10:31.208967+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:11.115107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202780+0000","last_change":"2024-04-27T01:09:57.996524+0000","last_active":"2024-04-27T01:10:31.202780+0000","last_peered":"2024-04-27T01:10:31.202780+0000","last_clean":"2024-04-27T01:10:31.202780+0000","last_became_active":"2024-04-27T01:09:57.996163+0000","last_became_peered":"2024-04-27T01:09:57.996163+0000","last_unstale":"2024-04-27T01:10:31.202780+0000","last_undegraded":"2024-04-27T01:10:31.202780+0000","last_fullsized":"2024-04-27T01:10:31.202780+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:50.034133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":44,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193538+0000","last_change":"2024-04-27T01:09:54.974415+0000","last_active":"2024-04-27T01:10:31.193538+0000","last_peered":"2024-04-27T01:10:31.193538+0000","last_clean":"2024-04-27T01:10:31.193538+0000","last_became_active":"2024-04-27T01:09:50.967576+0000","last_became_peered":"2024-04-27T01:09:50.967576+0000","last_unstale":"2024-04-27T01:10:31.193538+0000","last_undegraded":"2024-04-27T01:10:31.193538+0000","last_fullsized":"2024-04-27T01:10:31.193538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_clean_scrub_stamp":"2024-04-27T01:09:49.954065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:20.524736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030547220000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028778+0000","last_change":"2024-04-27T01:09:51.978554+0000","last_active":"2024-04-27T01:10:31.028778+0000","last_peered":"2024-04-27T01:10:31.028778+0000","last_clean":"2024-04-27T01:10:31.028778+0000","last_became_active":"2024-04-27T01:09:51.977736+0000","last_became_peered":"2024-04-27T01:09:51.977736+0000","last_unstale":"2024-04-27T01:10:31.028778+0000","last_undegraded":"2024-04-27T01:10:31.028778+0000","last_fullsized":"2024-04-27T01:10:31.028778+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:42.662246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'34","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040250+0000","last_change":"2024-04-27T01:09:53.984454+0000","last_active":"2024-04-27T01:11:02.040250+0000","last_peered":"2024-04-27T01:11:02.040250+0000","last_clean":"2024-04-27T01:11:02.040250+0000","last_became_active":"2024-04-27T01:09:53.983985+0000","last_became_peered":"2024-04-27T01:09:53.983985+0000","last_unstale":"2024-04-27T01:11:02.040250+0000","last_undegraded":"2024-04-27T01:11:02.040250+0000","last_fullsized":"2024-04-27T01:11:02.040250+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:13:37.096503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.208156+0000","last_change":"2024-04-27T01:09:55.990001+0000","last_active":"2024-04-27T01:10:31.208156+0000","last_peered":"2024-04-27T01:10:31.208156+0000","last_clean":"2024-04-27T01:10:31.208156+0000","last_became_active":"2024-04-27T01:09:55.989223+0000","last_became_peered":"2024-04-27T01:09:55.989223+0000","last_unstale":"2024-04-27T01:10:31.208156+0000","last_undegraded":"2024-04-27T01:10:31.208156+0000","last_fullsized":"2024-04-27T01:10:31.208156+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:20.167194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195036+0000","last_change":"2024-04-27T01:09:58.491893+0000","last_active":"2024-04-27T01:10:31.195036+0000","last_peered":"2024-04-27T01:10:31.195036+0000","last_clean":"2024-04-27T01:10:31.195036+0000","last_became_active":"2024-04-27T01:09:58.491600+0000","last_became_peered":"2024-04-27T01:09:58.491600+0000","last_unstale":"2024-04-27T01:10:31.195036+0000","last_undegraded":"2024-04-27T01:10:31.195036+0000","last_fullsized":"2024-04-27T01:10:31.195036+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:24:50.452967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.039268+0000","last_change":"2024-04-27T01:09:51.978284+0000","last_active":"2024-04-27T01:10:31.039268+0000","last_peered":"2024-04-27T01:10:31.039268+0000","last_clean":"2024-04-27T01:10:31.039268+0000","last_became_active":"2024-04-27T01:09:51.977919+0000","last_became_peered":"2024-04-27T01:09:51.977919+0000","last_unstale":"2024-04-27T01:10:31.039268+0000","last_undegraded":"2024-04-27T01:10:31.039268+0000","last_fullsized":"2024-04-27T01:10:31.039268+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:35:48.115350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029613+0000","last_change":"2024-04-27T01:09:54.971402+0000","last_active":"2024-04-27T01:10:31.029613+0000","last_peered":"2024-04-27T01:10:31.029613+0000","last_clean":"2024-04-27T01:10:31.029613+0000","last_became_active":"2024-04-27T01:09:50.970061+0000","last_became_peered":"2024-04-27T01:09:50.970061+0000","last_unstale":"2024-04-27T01:10:31.029613+0000","last_undegraded":"2024-04-27T01:10:31.029613+0000","last_fullsized":"2024-04-27T01:10:31.029613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_clean_scrub_stamp":"2024-04-27T01:09:49.954065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:22:48.313387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079872600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"69'19","reported_seq":68,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.038723+0000","last_change":"2024-04-27T01:09:53.984841+0000","last_active":"2024-04-27T01:10:31.038723+0000","last_peered":"2024-04-27T01:10:31.038723+0000","last_clean":"2024-04-27T01:10:31.038723+0000","last_became_active":"2024-04-27T01:09:53.984456+0000","last_became_peered":"2024-04-27T01:09:53.984456+0000","last_unstale":"2024-04-27T01:10:31.038723+0000","last_undegraded":"2024-04-27T01:10:31.038723+0000","last_fullsized":"2024-04-27T01:10:31.038723+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:15.182357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.030551+0000","last_change":"2024-04-27T01:09:55.989859+0000","last_active":"2024-04-27T01:10:31.030551+0000","last_peered":"2024-04-27T01:10:31.030551+0000","last_clean":"2024-04-27T01:10:31.030551+0000","last_became_active":"2024-04-27T01:09:55.989431+0000","last_became_peered":"2024-04-27T01:09:55.989431+0000","last_unstale":"2024-04-27T01:10:31.030551+0000","last_undegraded":"2024-04-27T01:10:31.030551+0000","last_fullsized":"2024-04-27T01:10:31.030551+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:45.939447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195032+0000","last_change":"2024-04-27T01:09:57.998807+0000","last_active":"2024-04-27T01:10:31.195032+0000","last_peered":"2024-04-27T01:10:31.195032+0000","last_clean":"2024-04-27T01:10:31.195032+0000","last_became_active":"2024-04-27T01:09:57.998457+0000","last_became_peered":"2024-04-27T01:09:57.998457+0000","last_unstale":"2024-04-27T01:10:31.195032+0000","last_undegraded":"2024-04-27T01:10:31.195032+0000","last_fullsized":"2024-04-27T01:10:31.195032+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:34:31.015288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'128","reported_seq":278,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:33.341976+0000","last_change":"2024-04-27T01:09:41.777614+0000","last_active":"2024-04-27T01:10:33.341976+0000","last_peered":"2024-04-27T01:10:33.341976+0000","last_clean":"2024-04-27T01:10:33.341976+0000","last_became_active":"2024-04-27T01:09:41.777074+0000","last_became_peered":"2024-04-27T01:09:41.777074+0000","last_unstale":"2024-04-27T01:10:33.341976+0000","last_undegraded":"2024-04-27T01:10:33.341976+0000","last_fullsized":"2024-04-27T01:10:33.341976+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:07:37.632381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:07:37.632381+0000","last_clean_scrub_stamp":"2024-04-27T01:07:37.632381+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:52.945958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200573+0000","last_change":"2024-04-27T01:09:51.978521+0000","last_active":"2024-04-27T01:10:31.200573+0000","last_peered":"2024-04-27T01:10:31.200573+0000","last_clean":"2024-04-27T01:10:31.200573+0000","last_became_active":"2024-04-27T01:09:51.978240+0000","last_became_peered":"2024-04-27T01:09:51.978240+0000","last_unstale":"2024-04-27T01:10:31.200573+0000","last_undegraded":"2024-04-27T01:10:31.200573+0000","last_fullsized":"2024-04-27T01:10:31.200573+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:42.911721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":67,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040625+0000","last_change":"2024-04-27T01:09:54.096643+0000","last_active":"2024-04-27T01:11:02.040625+0000","last_peered":"2024-04-27T01:11:02.040625+0000","last_clean":"2024-04-27T01:11:02.040625+0000","last_became_active":"2024-04-27T01:09:54.096283+0000","last_became_peered":"2024-04-27T01:09:54.096283+0000","last_unstale":"2024-04-27T01:11:02.040625+0000","last_undegraded":"2024-04-27T01:11:02.040625+0000","last_fullsized":"2024-04-27T01:11:02.040625+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:15:47.195945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195207+0000","last_change":"2024-04-27T01:09:55.987753+0000","last_active":"2024-04-27T01:10:31.195207+0000","last_peered":"2024-04-27T01:10:31.195207+0000","last_clean":"2024-04-27T01:10:31.195207+0000","last_became_active":"2024-04-27T01:09:55.987000+0000","last_became_peered":"2024-04-27T01:09:55.987000+0000","last_unstale":"2024-04-27T01:10:31.195207+0000","last_undegraded":"2024-04-27T01:10:31.195207+0000","last_fullsized":"2024-04-27T01:10:31.195207+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:18.394267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.039496+0000","last_change":"2024-04-27T01:09:57.997995+0000","last_active":"2024-04-27T01:10:31.039496+0000","last_peered":"2024-04-27T01:10:31.039496+0000","last_clean":"2024-04-27T01:10:31.039496+0000","last_became_active":"2024-04-27T01:09:57.997664+0000","last_became_peered":"2024-04-27T01:09:57.997664+0000","last_unstale":"2024-04-27T01:10:31.039496+0000","last_undegraded":"2024-04-27T01:10:31.039496+0000","last_fullsized":"2024-04-27T01:10:31.039496+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:04.460907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"62'1","reported_seq":39,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202595+0000","last_change":"2024-04-27T01:09:51.975394+0000","last_active":"2024-04-27T01:10:31.202595+0000","last_peered":"2024-04-27T01:10:31.202595+0000","last_clean":"2024-04-27T01:10:31.202595+0000","last_became_active":"2024-04-27T01:09:51.974515+0000","last_became_peered":"2024-04-27T01:09:51.974515+0000","last_unstale":"2024-04-27T01:10:31.202595+0000","last_undegraded":"2024-04-27T01:10:31.202595+0000","last_fullsized":"2024-04-27T01:10:31.202595+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:28:50.723732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":112,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:05.357963+0000","last_change":"2024-04-27T01:09:54.974030+0000","last_active":"2024-04-27T01:11:05.357963+0000","last_peered":"2024-04-27T01:11:05.357963+0000","last_clean":"2024-04-27T01:11:05.357963+0000","last_became_active":"2024-04-27T01:09:50.967643+0000","last_became_peered":"2024-04-27T01:09:50.967643+0000","last_unstale":"2024-04-27T01:11:05.357963+0000","last_undegraded":"2024-04-27T01:11:05.357963+0000","last_fullsized":"2024-04-27T01:11:05.357963+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_clean_scrub_stamp":"2024-04-27T01:09:49.954065+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:02:23.146239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000441765,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":67,"num_read_kb":62,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":47,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.208084+0000","last_change":"2024-04-27T01:09:53.981423+0000","last_active":"2024-04-27T01:10:31.208084+0000","last_peered":"2024-04-27T01:10:31.208084+0000","last_clean":"2024-04-27T01:10:31.208084+0000","last_became_active":"2024-04-27T01:09:53.980996+0000","last_became_peered":"2024-04-27T01:09:53.980996+0000","last_unstale":"2024-04-27T01:10:31.208084+0000","last_undegraded":"2024-04-27T01:10:31.208084+0000","last_fullsized":"2024-04-27T01:10:31.208084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:37.319973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195444+0000","last_change":"2024-04-27T01:09:55.994152+0000","last_active":"2024-04-27T01:10:31.195444+0000","last_peered":"2024-04-27T01:10:31.195444+0000","last_clean":"2024-04-27T01:10:31.195444+0000","last_became_active":"2024-04-27T01:09:55.993872+0000","last_became_peered":"2024-04-27T01:09:55.993872+0000","last_unstale":"2024-04-27T01:10:31.195444+0000","last_undegraded":"2024-04-27T01:10:31.195444+0000","last_fullsized":"2024-04-27T01:10:31.195444+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:11:49.192838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.033302+0000","last_change":"2024-04-27T01:09:58.491368+0000","last_active":"2024-04-27T01:10:31.033302+0000","last_peered":"2024-04-27T01:10:31.033302+0000","last_clean":"2024-04-27T01:10:31.033302+0000","last_became_active":"2024-04-27T01:09:58.490925+0000","last_became_peered":"2024-04-27T01:09:58.490925+0000","last_unstale":"2024-04-27T01:10:31.033302+0000","last_undegraded":"2024-04-27T01:10:31.033302+0000","last_fullsized":"2024-04-27T01:10:31.033302+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:53.417660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.039810+0000","last_change":"2024-04-27T01:09:51.979283+0000","last_active":"2024-04-27T01:10:31.039810+0000","last_peered":"2024-04-27T01:10:31.039810+0000","last_clean":"2024-04-27T01:10:31.039810+0000","last_became_active":"2024-04-27T01:09:51.978963+0000","last_became_peered":"2024-04-27T01:09:51.978963+0000","last_unstale":"2024-04-27T01:10:31.039810+0000","last_undegraded":"2024-04-27T01:10:31.039810+0000","last_fullsized":"2024-04-27T01:10:31.039810+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:58:27.383762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"69'25","reported_seq":77,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.208764+0000","last_change":"2024-04-27T01:09:54.096966+0000","last_active":"2024-04-27T01:10:31.208764+0000","last_peered":"2024-04-27T01:10:31.208764+0000","last_clean":"2024-04-27T01:10:31.208764+0000","last_became_active":"2024-04-27T01:09:54.096116+0000","last_became_peered":"2024-04-27T01:09:54.096116+0000","last_unstale":"2024-04-27T01:10:31.208764+0000","last_undegraded":"2024-04-27T01:10:31.208764+0000","last_fullsized":"2024-04-27T01:10:31.208764+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:11:54.089352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194466+0000","last_change":"2024-04-27T01:09:55.985443+0000","last_active":"2024-04-27T01:10:31.194466+0000","last_peered":"2024-04-27T01:10:31.194466+0000","last_clean":"2024-04-27T01:10:31.194466+0000","last_became_active":"2024-04-27T01:09:55.984939+0000","last_became_peered":"2024-04-27T01:09:55.984939+0000","last_unstale":"2024-04-27T01:10:31.194466+0000","last_undegraded":"2024-04-27T01:10:31.194466+0000","last_fullsized":"2024-04-27T01:10:31.194466+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:25.407393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202216+0000","last_change":"2024-04-27T01:09:58.004290+0000","last_active":"2024-04-27T01:10:31.202216+0000","last_peered":"2024-04-27T01:10:31.202216+0000","last_clean":"2024-04-27T01:10:31.202216+0000","last_became_active":"2024-04-27T01:09:58.003641+0000","last_became_peered":"2024-04-27T01:09:58.003641+0000","last_unstale":"2024-04-27T01:10:31.202216+0000","last_undegraded":"2024-04-27T01:10:31.202216+0000","last_fullsized":"2024-04-27T01:10:31.202216+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:05.432205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"62'1","reported_seq":39,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194341+0000","last_change":"2024-04-27T01:09:51.975079+0000","last_active":"2024-04-27T01:10:31.194341+0000","last_peered":"2024-04-27T01:10:31.194341+0000","last_clean":"2024-04-27T01:10:31.194341+0000","last_became_active":"2024-04-27T01:09:51.974755+0000","last_became_peered":"2024-04-27T01:09:51.974755+0000","last_unstale":"2024-04-27T01:10:31.194341+0000","last_undegraded":"2024-04-27T01:10:31.194341+0000","last_fullsized":"2024-04-27T01:10:31.194341+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:23:00.685692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":56,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.034130+0000","last_change":"2024-04-27T01:09:53.982529+0000","last_active":"2024-04-27T01:10:31.034130+0000","last_peered":"2024-04-27T01:10:31.034130+0000","last_clean":"2024-04-27T01:10:31.034130+0000","last_became_active":"2024-04-27T01:09:53.982188+0000","last_became_peered":"2024-04-27T01:09:53.982188+0000","last_unstale":"2024-04-27T01:10:31.034130+0000","last_undegraded":"2024-04-27T01:10:31.034130+0000","last_fullsized":"2024-04-27T01:10:31.034130+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:52.168079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":49,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040279+0000","last_change":"2024-04-27T01:09:55.986183+0000","last_active":"2024-04-27T01:11:02.040279+0000","last_peered":"2024-04-27T01:11:02.040279+0000","last_clean":"2024-04-27T01:11:02.040279+0000","last_became_active":"2024-04-27T01:09:55.985776+0000","last_became_peered":"2024-04-27T01:09:55.985776+0000","last_unstale":"2024-04-27T01:11:02.040279+0000","last_undegraded":"2024-04-27T01:11:02.040279+0000","last_fullsized":"2024-04-27T01:11:02.040279+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:13.714957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193722+0000","last_change":"2024-04-27T01:09:58.493484+0000","last_active":"2024-04-27T01:10:31.193722+0000","last_peered":"2024-04-27T01:10:31.193722+0000","last_clean":"2024-04-27T01:10:31.193722+0000","last_became_active":"2024-04-27T01:09:58.493199+0000","last_became_peered":"2024-04-27T01:09:58.493199+0000","last_unstale":"2024-04-27T01:10:31.193722+0000","last_undegraded":"2024-04-27T01:10:31.193722+0000","last_fullsized":"2024-04-27T01:10:31.193722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:14.831100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.209142+0000","last_change":"2024-04-27T01:09:51.972465+0000","last_active":"2024-04-27T01:10:31.209142+0000","last_peered":"2024-04-27T01:10:31.209142+0000","last_clean":"2024-04-27T01:10:31.209142+0000","last_became_active":"2024-04-27T01:09:51.971917+0000","last_became_peered":"2024-04-27T01:09:51.971917+0000","last_unstale":"2024-04-27T01:10:31.209142+0000","last_undegraded":"2024-04-27T01:10:31.209142+0000","last_fullsized":"2024-04-27T01:10:31.209142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:42.551048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":62,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029172+0000","last_change":"2024-04-27T01:09:53.977524+0000","last_active":"2024-04-27T01:10:31.029172+0000","last_peered":"2024-04-27T01:10:31.029172+0000","last_clean":"2024-04-27T01:10:31.029172+0000","last_became_active":"2024-04-27T01:09:53.977141+0000","last_became_peered":"2024-04-27T01:09:53.977141+0000","last_unstale":"2024-04-27T01:10:31.029172+0000","last_undegraded":"2024-04-27T01:10:31.029172+0000","last_fullsized":"2024-04-27T01:10:31.029172+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:24.476172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":52,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040298+0000","last_change":"2024-04-27T01:09:55.994468+0000","last_active":"2024-04-27T01:11:02.040298+0000","last_peered":"2024-04-27T01:11:02.040298+0000","last_clean":"2024-04-27T01:11:02.040298+0000","last_became_active":"2024-04-27T01:09:55.994143+0000","last_became_peered":"2024-04-27T01:09:55.994143+0000","last_unstale":"2024-04-27T01:11:02.040298+0000","last_undegraded":"2024-04-27T01:11:02.040298+0000","last_fullsized":"2024-04-27T01:11:02.040298+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:43:23.714459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028520+0000","last_change":"2024-04-27T01:09:57.996696+0000","last_active":"2024-04-27T01:10:31.028520+0000","last_peered":"2024-04-27T01:10:31.028520+0000","last_clean":"2024-04-27T01:10:31.028520+0000","last_became_active":"2024-04-27T01:09:57.996331+0000","last_became_peered":"2024-04-27T01:09:57.996331+0000","last_unstale":"2024-04-27T01:10:31.028520+0000","last_undegraded":"2024-04-27T01:10:31.028520+0000","last_fullsized":"2024-04-27T01:10:31.028520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:05:40.417633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202647+0000","last_change":"2024-04-27T01:09:51.981032+0000","last_active":"2024-04-27T01:10:31.202647+0000","last_peered":"2024-04-27T01:10:31.202647+0000","last_clean":"2024-04-27T01:10:31.202647+0000","last_became_active":"2024-04-27T01:09:51.980554+0000","last_became_peered":"2024-04-27T01:09:51.980554+0000","last_unstale":"2024-04-27T01:10:31.202647+0000","last_undegraded":"2024-04-27T01:10:31.202647+0000","last_fullsized":"2024-04-27T01:10:31.202647+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:07:13.815364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'30","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200900+0000","last_change":"2024-04-27T01:09:54.096814+0000","last_active":"2024-04-27T01:10:31.200900+0000","last_peered":"2024-04-27T01:10:31.200900+0000","last_clean":"2024-04-27T01:10:31.200900+0000","last_became_active":"2024-04-27T01:09:54.096423+0000","last_became_peered":"2024-04-27T01:09:54.096423+0000","last_unstale":"2024-04-27T01:10:31.200900+0000","last_undegraded":"2024-04-27T01:10:31.200900+0000","last_fullsized":"2024-04-27T01:10:31.200900+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:53:25.325208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028454+0000","last_change":"2024-04-27T01:09:55.992285+0000","last_active":"2024-04-27T01:10:31.028454+0000","last_peered":"2024-04-27T01:10:31.028454+0000","last_clean":"2024-04-27T01:10:31.028454+0000","last_became_active":"2024-04-27T01:09:55.992003+0000","last_became_peered":"2024-04-27T01:09:55.992003+0000","last_unstale":"2024-04-27T01:10:31.028454+0000","last_undegraded":"2024-04-27T01:10:31.028454+0000","last_fullsized":"2024-04-27T01:10:31.028454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:29:04.778310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194046+0000","last_change":"2024-04-27T01:09:58.003940+0000","last_active":"2024-04-27T01:10:31.194046+0000","last_peered":"2024-04-27T01:10:31.194046+0000","last_clean":"2024-04-27T01:10:31.194046+0000","last_became_active":"2024-04-27T01:09:58.003730+0000","last_became_peered":"2024-04-27T01:09:58.003730+0000","last_unstale":"2024-04-27T01:10:31.194046+0000","last_undegraded":"2024-04-27T01:10:31.194046+0000","last_fullsized":"2024-04-27T01:10:31.194046+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:50:57.443520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":49,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040734+0000","last_change":"2024-04-27T01:09:55.986323+0000","last_active":"2024-04-27T01:11:02.040734+0000","last_peered":"2024-04-27T01:11:02.040734+0000","last_clean":"2024-04-27T01:11:02.040734+0000","last_became_active":"2024-04-27T01:09:55.986022+0000","last_became_peered":"2024-04-27T01:09:55.986022+0000","last_unstale":"2024-04-27T01:11:02.040734+0000","last_undegraded":"2024-04-27T01:11:02.040734+0000","last_fullsized":"2024-04-27T01:11:02.040734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:18:53.897875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200812+0000","last_change":"2024-04-27T01:09:51.974389+0000","last_active":"2024-04-27T01:10:31.200812+0000","last_peered":"2024-04-27T01:10:31.200812+0000","last_clean":"2024-04-27T01:10:31.200812+0000","last_became_active":"2024-04-27T01:09:51.974094+0000","last_became_peered":"2024-04-27T01:09:51.974094+0000","last_unstale":"2024-04-27T01:10:31.200812+0000","last_undegraded":"2024-04-27T01:10:31.200812+0000","last_fullsized":"2024-04-27T01:10:31.200812+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:49:15.151744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.038572+0000","last_change":"2024-04-27T01:09:53.983723+0000","last_active":"2024-04-27T01:10:31.038572+0000","last_peered":"2024-04-27T01:10:31.038572+0000","last_clean":"2024-04-27T01:10:31.038572+0000","last_became_active":"2024-04-27T01:09:53.983380+0000","last_became_peered":"2024-04-27T01:09:53.983380+0000","last_unstale":"2024-04-27T01:10:31.038572+0000","last_undegraded":"2024-04-27T01:10:31.038572+0000","last_fullsized":"2024-04-27T01:10:31.038572+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:48.838115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200964+0000","last_change":"2024-04-27T01:09:58.492072+0000","last_active":"2024-04-27T01:10:31.200964+0000","last_peered":"2024-04-27T01:10:31.200964+0000","last_clean":"2024-04-27T01:10:31.200964+0000","last_became_active":"2024-04-27T01:09:58.491675+0000","last_became_peered":"2024-04-27T01:09:58.491675+0000","last_unstale":"2024-04-27T01:10:31.200964+0000","last_undegraded":"2024-04-27T01:10:31.200964+0000","last_fullsized":"2024-04-27T01:10:31.200964+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:19:27.606788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195371+0000","last_change":"2024-04-27T01:09:55.990732+0000","last_active":"2024-04-27T01:10:31.195371+0000","last_peered":"2024-04-27T01:10:31.195371+0000","last_clean":"2024-04-27T01:10:31.195371+0000","last_became_active":"2024-04-27T01:09:55.990078+0000","last_became_peered":"2024-04-27T01:09:55.990078+0000","last_unstale":"2024-04-27T01:10:31.195371+0000","last_undegraded":"2024-04-27T01:10:31.195371+0000","last_fullsized":"2024-04-27T01:10:31.195371+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:15:04.385756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028929+0000","last_change":"2024-04-27T01:09:51.974876+0000","last_active":"2024-04-27T01:10:31.028929+0000","last_peered":"2024-04-27T01:10:31.028929+0000","last_clean":"2024-04-27T01:10:31.028929+0000","last_became_active":"2024-04-27T01:09:51.974577+0000","last_became_peered":"2024-04-27T01:09:51.974577+0000","last_unstale":"2024-04-27T01:10:31.028929+0000","last_undegraded":"2024-04-27T01:10:31.028929+0000","last_fullsized":"2024-04-27T01:10:31.028929+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:06:31.526042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'17","reported_seq":60,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029141+0000","last_change":"2024-04-27T01:09:53.981121+0000","last_active":"2024-04-27T01:10:31.029141+0000","last_peered":"2024-04-27T01:10:31.029141+0000","last_clean":"2024-04-27T01:10:31.029141+0000","last_became_active":"2024-04-27T01:09:53.980643+0000","last_became_peered":"2024-04-27T01:09:53.980643+0000","last_unstale":"2024-04-27T01:10:31.029141+0000","last_undegraded":"2024-04-27T01:10:31.029141+0000","last_fullsized":"2024-04-27T01:10:31.029141+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:09.835870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.030903+0000","last_change":"2024-04-27T01:09:58.001560+0000","last_active":"2024-04-27T01:10:31.030903+0000","last_peered":"2024-04-27T01:10:31.030903+0000","last_clean":"2024-04-27T01:10:31.030903+0000","last_became_active":"2024-04-27T01:09:58.001241+0000","last_became_peered":"2024-04-27T01:09:58.001241+0000","last_unstale":"2024-04-27T01:10:31.030903+0000","last_undegraded":"2024-04-27T01:10:31.030903+0000","last_fullsized":"2024-04-27T01:10:31.030903+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:15:24.574837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.040017+0000","last_change":"2024-04-27T01:09:55.994476+0000","last_active":"2024-04-27T01:10:31.040017+0000","last_peered":"2024-04-27T01:10:31.040017+0000","last_clean":"2024-04-27T01:10:31.040017+0000","last_became_active":"2024-04-27T01:09:55.993751+0000","last_became_peered":"2024-04-27T01:09:55.993751+0000","last_unstale":"2024-04-27T01:10:31.040017+0000","last_undegraded":"2024-04-27T01:10:31.040017+0000","last_fullsized":"2024-04-27T01:10:31.040017+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:57.091034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194581+0000","last_change":"2024-04-27T01:09:51.975073+0000","last_active":"2024-04-27T01:10:31.194581+0000","last_peered":"2024-04-27T01:10:31.194581+0000","last_clean":"2024-04-27T01:10:31.194581+0000","last_became_active":"2024-04-27T01:09:51.974368+0000","last_became_peered":"2024-04-27T01:09:51.974368+0000","last_unstale":"2024-04-27T01:10:31.194581+0000","last_undegraded":"2024-04-27T01:10:31.194581+0000","last_fullsized":"2024-04-27T01:10:31.194581+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:40:27.867034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":81,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029257+0000","last_change":"2024-04-27T01:10:22.475719+0000","last_active":"2024-04-27T01:10:31.029257+0000","last_peered":"2024-04-27T01:10:31.029257+0000","last_clean":"2024-04-27T01:10:31.029257+0000","last_became_active":"2024-04-27T01:10:22.468557+0000","last_became_peered":"2024-04-27T01:10:22.468557+0000","last_unstale":"2024-04-27T01:10:31.029257+0000","last_undegraded":"2024-04-27T01:10:31.029257+0000","last_fullsized":"2024-04-27T01:10:31.029257+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:59:47.022142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029500+0000","last_change":"2024-04-27T01:09:57.994354+0000","last_active":"2024-04-27T01:10:31.029500+0000","last_peered":"2024-04-27T01:10:31.029500+0000","last_clean":"2024-04-27T01:10:31.029500+0000","last_became_active":"2024-04-27T01:09:57.994072+0000","last_became_peered":"2024-04-27T01:09:57.994072+0000","last_unstale":"2024-04-27T01:10:31.029500+0000","last_undegraded":"2024-04-27T01:10:31.029500+0000","last_fullsized":"2024-04-27T01:10:31.029500+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:56:45.195389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":49,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040265+0000","last_change":"2024-04-27T01:09:55.989649+0000","last_active":"2024-04-27T01:11:02.040265+0000","last_peered":"2024-04-27T01:11:02.040265+0000","last_clean":"2024-04-27T01:11:02.040265+0000","last_became_active":"2024-04-27T01:09:55.989303+0000","last_became_peered":"2024-04-27T01:09:55.989303+0000","last_unstale":"2024-04-27T01:11:02.040265+0000","last_undegraded":"2024-04-27T01:11:02.040265+0000","last_fullsized":"2024-04-27T01:11:02.040265+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:32:17.454412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202271+0000","last_change":"2024-04-27T01:09:51.978202+0000","last_active":"2024-04-27T01:10:31.202271+0000","last_peered":"2024-04-27T01:10:31.202271+0000","last_clean":"2024-04-27T01:10:31.202271+0000","last_became_active":"2024-04-27T01:09:51.977897+0000","last_became_peered":"2024-04-27T01:09:51.977897+0000","last_unstale":"2024-04-27T01:10:31.202271+0000","last_undegraded":"2024-04-27T01:10:31.202271+0000","last_fullsized":"2024-04-27T01:10:31.202271+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:39:39.844375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":44,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028807+0000","last_change":"2024-04-27T01:09:53.980407+0000","last_active":"2024-04-27T01:10:31.028807+0000","last_peered":"2024-04-27T01:10:31.028807+0000","last_clean":"2024-04-27T01:10:31.028807+0000","last_became_active":"2024-04-27T01:09:53.980029+0000","last_became_peered":"2024-04-27T01:09:53.980029+0000","last_unstale":"2024-04-27T01:10:31.028807+0000","last_undegraded":"2024-04-27T01:10:31.028807+0000","last_fullsized":"2024-04-27T01:10:31.028807+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:02.744677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":62,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201300,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32468,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712684,"statfs":{"total":95995035648,"available":95961788416,"internally_reserved":0,"allocated":1323008,"data_stored":874698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32444,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712708,"statfs":{"total":95995035648,"available":95961812992,"internally_reserved":0,"allocated":1298432,"data_stored":873600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":284687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712676,"statfs":{"total":95995035648,"available":95961780224,"internally_reserved":0,"allocated":1327104,"data_stored":875387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":716800,"data_stored":284049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":753664,"data_stored":285565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":737280,"data_stored":284830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724573,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":716800,"data_stored":284137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T01:11:11.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-27T01:11:11.979 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:11 smithi162 ceph-mon[26714]: from='client.14763 v1:172.21.15.162:0/2917863650' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:11:11.979 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:11 smithi162 ceph-mon[37680]: from='client.14763 v1:172.21.15.162:0/2917863650' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:11:12.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:11 smithi181 ceph-mon[30237]: from='client.14763 v1:172.21.15.162:0/2917863650' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:11:12.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:11:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:11:12.983 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:12 smithi162 ceph-mon[37680]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:12.983 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:12 smithi162 ceph-mon[26714]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:12 smithi181 ceph-mon[30237]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:13.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:13.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:14.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:14.248 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:11:15.004 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:14 smithi162 ceph-mon[37680]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:15.004 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:14 smithi162 ceph-mon[26714]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:15.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:14 smithi181 ceph-mon[30237]: pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:15.589 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:15 smithi162 systemd[1]: Starting Ceph alertmanager.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:11:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.424+0000 7f43e793b700 1 -- 172.21.15.162:0/2412701133 >> v1:172.21.15.162:6790/0 conn(0x7f43e01152e0 legacy=0x7f43e01176d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.424+0000 7f43e793b700 1 -- 172.21.15.162:0/2412701133 shutdown_connections 2024-04-27T01:11:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.424+0000 7f43e793b700 1 -- 172.21.15.162:0/2412701133 >> 172.21.15.162:0/2412701133 conn(0x7f43e00fca30 msgr2=0x7f43e00fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.424+0000 7f43e793b700 1 -- 172.21.15.162:0/2412701133 shutdown_connections 2024-04-27T01:11:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.424+0000 7f43e793b700 1 -- 172.21.15.162:0/2412701133 wait complete. 2024-04-27T01:11:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43e793b700 1 Processor -- start 2024-04-27T01:11:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43e793b700 1 -- start start 2024-04-27T01:11:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43e793b700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43e01ad660 con 0x7f43e0100fd0 2024-04-27T01:11:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43e793b700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43e01ae880 con 0x7f43e0106dd0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43e793b700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f43e01afac0 con 0x7f43e01152e0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43e56d7700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f43e0100fd0 0x7f43e0114a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:41236/0 (socket says 172.21.15.162:41236) 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43e56d7700 1 -- 172.21.15.162:0/3064791565 learned_addr learned my addr 172.21.15.162:0/3064791565 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614182775 0 0) 0x7f43e01afac0 con 0x7f43e01152e0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43cc0031f0 con 0x7f43e01152e0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2114467532 0 0) 0x7f43e01ad660 con 0x7f43e0100fd0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43e01afac0 con 0x7f43e0100fd0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3608143972 0 0) 0x7f43e01ae880 con 0x7f43e0106dd0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43e01ad660 con 0x7f43e0106dd0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4091399733 0 0) 0x7f43cc0031f0 con 0x7f43e01152e0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43e01ae880 con 0x7f43e01152e0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f43d8004880 con 0x7f43e01152e0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.425+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 882836726 0 0) 0x7f43e01afac0 con 0x7f43e0100fd0 2024-04-27T01:11:16.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43cc0031f0 con 0x7f43e0100fd0 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f43dc0018f0 con 0x7f43e0100fd0 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2450418395 0 0) 0x7f43e01ad660 con 0x7f43e0106dd0 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f43e01afac0 con 0x7f43e0106dd0 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f43d0001a30 con 0x7f43e0106dd0 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 604300988 0 0) 0x7f43e01ae880 con 0x7f43e01152e0 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 >> v1:172.21.15.181:6789/0 conn(0x7f43e0106dd0 legacy=0x7f43e01a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 >> v1:172.21.15.162:6789/0 conn(0x7f43e0100fd0 legacy=0x7f43e0114a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e01b0d00 con 0x7f43e01152e0 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43e793b700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f43e01ad890 con 0x7f43e01152e0 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43e793b700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f43e01add50 con 0x7f43e01152e0 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f43d8001e30 con 0x7f43e01152e0 2024-04-27T01:11:16.418 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.426+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f43d8005270 con 0x7f43e01152e0 2024-04-27T01:11:16.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.427+0000 7f43e793b700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c4005120 con 0x7f43e01152e0 2024-04-27T01:11:16.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.427+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f43d8003a90 con 0x7f43e01152e0 2024-04-27T01:11:16.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.428+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7f43d80886e0 con 0x7f43e01152e0 2024-04-27T01:11:16.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.433+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f43d8001fe0 con 0x7f43e01152e0 2024-04-27T01:11:16.589 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:16 smithi162 podman[72139]: 2024-04-27 01:11:16.460057455 +0000 UTC m=+0.248827911 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-04-27T01:11:16.589 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:16 smithi162 podman[72139]: 2024-04-27 01:11:16.565483208 +0000 UTC m=+0.354253654 volume create 6ad03e9c425108c5ad494939eb382ed9f5929ae4d0c19d7464033af8e547cede 2024-04-27T01:11:16.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.632+0000 7f43e793b700 1 -- 172.21.15.162:0/3064791565 --> v1:172.21.15.162:6800/624703537 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f43c4000b90 con 0x7f43cc06f020 2024-04-27T01:11:16.631 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.642+0000 7f43d67fc700 1 -- 172.21.15.162:0/3064791565 <== mgr.14622 v1:172.21.15.162:6800/624703537 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346507 (unknown 2965378022 0 2726796132) 0x7f43c4000b90 con 0x7f43cc06f020 2024-04-27T01:11:16.635 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:11:16.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.645+0000 7f43e793b700 1 -- 172.21.15.162:0/3064791565 >> v1:172.21.15.162:6800/624703537 conn(0x7f43cc06f020 legacy=0x7f43cc0714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:16.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.645+0000 7f43e793b700 1 -- 172.21.15.162:0/3064791565 >> v1:172.21.15.162:6790/0 conn(0x7f43e01152e0 legacy=0x7f43e01abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:16.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.645+0000 7f43e793b700 1 -- 172.21.15.162:0/3064791565 shutdown_connections 2024-04-27T01:11:16.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.645+0000 7f43e793b700 1 -- 172.21.15.162:0/3064791565 >> 172.21.15.162:0/3064791565 conn(0x7f43e00fca30 msgr2=0x7f43e00fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:16.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.645+0000 7f43e793b700 1 -- 172.21.15.162:0/3064791565 shutdown_connections 2024-04-27T01:11:16.638 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:16.645+0000 7f43e793b700 1 -- 172.21.15.162:0/3064791565 wait complete. 2024-04-27T01:11:16.639 INFO:teuthology.orchestra.run.smithi162.stderr:dumped all 2024-04-27T01:11:16.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:16 smithi162 ceph-mon[37680]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:16.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:16.985 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:16 smithi162 ceph-mon[26714]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:16.985 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:16.985 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:16 smithi162 podman[72139]: 2024-04-27 01:11:16.674687679 +0000 UTC m=+0.463458126 container create 7c13e184d0c7282dd5cecf024cd1f6a969dfbacda025cb339094daf6c2c77da1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:17.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:16 smithi181 ceph-mon[30237]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:17.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:17.415 INFO:teuthology.orchestra.run.smithi162.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-04-27T01:11:15.083777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1294,"num_read_kb":1108,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":797,"ondisk_log_size":797,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":252724,"kb_used_data":7452,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749708492,"statfs":{"total":767960285184,"available":767701495808,"internally_reserved":0,"allocated":7630848,"data_stored":4046953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003178"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028470+0000","last_change":"2024-04-27T01:09:57.993694+0000","last_active":"2024-04-27T01:10:31.028470+0000","last_peered":"2024-04-27T01:10:31.028470+0000","last_clean":"2024-04-27T01:10:31.028470+0000","last_became_active":"2024-04-27T01:09:57.993370+0000","last_became_peered":"2024-04-27T01:09:57.993370+0000","last_unstale":"2024-04-27T01:10:31.028470+0000","last_undegraded":"2024-04-27T01:10:31.028470+0000","last_fullsized":"2024-04-27T01:10:31.028470+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:07.224904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.033410+0000","last_change":"2024-04-27T01:09:55.991919+0000","last_active":"2024-04-27T01:10:31.033410+0000","last_peered":"2024-04-27T01:10:31.033410+0000","last_clean":"2024-04-27T01:10:31.033410+0000","last_became_active":"2024-04-27T01:09:55.991263+0000","last_became_peered":"2024-04-27T01:09:55.991263+0000","last_unstale":"2024-04-27T01:10:31.033410+0000","last_undegraded":"2024-04-27T01:10:31.033410+0000","last_fullsized":"2024-04-27T01:10:31.033410+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:55:14.625326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.214033+0000","last_change":"2024-04-27T01:09:51.968754+0000","last_active":"2024-04-27T01:10:31.214033+0000","last_peered":"2024-04-27T01:10:31.214033+0000","last_clean":"2024-04-27T01:10:31.214033+0000","last_became_active":"2024-04-27T01:09:51.968457+0000","last_became_peered":"2024-04-27T01:09:51.968457+0000","last_unstale":"2024-04-27T01:10:31.214033+0000","last_undegraded":"2024-04-27T01:10:31.214033+0000","last_fullsized":"2024-04-27T01:10:31.214033+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:16:11.811934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'16","reported_seq":76,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028454+0000","last_change":"2024-04-27T01:10:22.096093+0000","last_active":"2024-04-27T01:10:31.028454+0000","last_peered":"2024-04-27T01:10:31.028454+0000","last_clean":"2024-04-27T01:10:31.028454+0000","last_became_active":"2024-04-27T01:10:22.082641+0000","last_became_peered":"2024-04-27T01:10:22.082641+0000","last_unstale":"2024-04-27T01:10:31.028454+0000","last_undegraded":"2024-04-27T01:10:31.028454+0000","last_fullsized":"2024-04-27T01:10:31.028454+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:44.047387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202083+0000","last_change":"2024-04-27T01:09:58.493006+0000","last_active":"2024-04-27T01:10:31.202083+0000","last_peered":"2024-04-27T01:10:31.202083+0000","last_clean":"2024-04-27T01:10:31.202083+0000","last_became_active":"2024-04-27T01:09:58.492609+0000","last_became_peered":"2024-04-27T01:09:58.492609+0000","last_unstale":"2024-04-27T01:10:31.202083+0000","last_undegraded":"2024-04-27T01:10:31.202083+0000","last_fullsized":"2024-04-27T01:10:31.202083+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:27.598635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028509+0000","last_change":"2024-04-27T01:09:55.988582+0000","last_active":"2024-04-27T01:10:31.028509+0000","last_peered":"2024-04-27T01:10:31.028509+0000","last_clean":"2024-04-27T01:10:31.028509+0000","last_became_active":"2024-04-27T01:09:55.988296+0000","last_became_peered":"2024-04-27T01:09:55.988296+0000","last_unstale":"2024-04-27T01:10:31.028509+0000","last_undegraded":"2024-04-27T01:10:31.028509+0000","last_fullsized":"2024-04-27T01:10:31.028509+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:54.291915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.203557+0000","last_change":"2024-04-27T01:09:51.977650+0000","last_active":"2024-04-27T01:10:31.203557+0000","last_peered":"2024-04-27T01:10:31.203557+0000","last_clean":"2024-04-27T01:10:31.203557+0000","last_became_active":"2024-04-27T01:09:51.977352+0000","last_became_peered":"2024-04-27T01:09:51.977352+0000","last_unstale":"2024-04-27T01:10:31.203557+0000","last_undegraded":"2024-04-27T01:10:31.203557+0000","last_fullsized":"2024-04-27T01:10:31.203557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:15.672832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'39","reported_seq":102,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.204399+0000","last_change":"2024-04-27T01:09:53.980284+0000","last_active":"2024-04-27T01:10:31.204399+0000","last_peered":"2024-04-27T01:10:31.204399+0000","last_clean":"2024-04-27T01:10:31.204399+0000","last_became_active":"2024-04-27T01:09:53.979869+0000","last_became_peered":"2024-04-27T01:09:53.979869+0000","last_unstale":"2024-04-27T01:10:31.204399+0000","last_undegraded":"2024-04-27T01:10:31.204399+0000","last_fullsized":"2024-04-27T01:10:31.204399+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:56.069731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202477+0000","last_change":"2024-04-27T01:09:58.002523+0000","last_active":"2024-04-27T01:10:31.202477+0000","last_peered":"2024-04-27T01:10:31.202477+0000","last_clean":"2024-04-27T01:10:31.202477+0000","last_became_active":"2024-04-27T01:09:58.002166+0000","last_became_peered":"2024-04-27T01:09:58.002166+0000","last_unstale":"2024-04-27T01:10:31.202477+0000","last_undegraded":"2024-04-27T01:10:31.202477+0000","last_fullsized":"2024-04-27T01:10:31.202477+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:39:05.197998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193278+0000","last_change":"2024-04-27T01:09:55.984659+0000","last_active":"2024-04-27T01:10:31.193278+0000","last_peered":"2024-04-27T01:10:31.193278+0000","last_clean":"2024-04-27T01:10:31.193278+0000","last_became_active":"2024-04-27T01:09:55.984163+0000","last_became_peered":"2024-04-27T01:09:55.984163+0000","last_unstale":"2024-04-27T01:10:31.193278+0000","last_undegraded":"2024-04-27T01:10:31.193278+0000","last_fullsized":"2024-04-27T01:10:31.193278+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:47.572049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194142+0000","last_change":"2024-04-27T01:09:51.975375+0000","last_active":"2024-04-27T01:10:31.194142+0000","last_peered":"2024-04-27T01:10:31.194142+0000","last_clean":"2024-04-27T01:10:31.194142+0000","last_became_active":"2024-04-27T01:09:51.975035+0000","last_became_peered":"2024-04-27T01:09:51.975035+0000","last_unstale":"2024-04-27T01:10:31.194142+0000","last_undegraded":"2024-04-27T01:10:31.194142+0000","last_fullsized":"2024-04-27T01:10:31.194142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:34.939844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'23","reported_seq":71,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028627+0000","last_change":"2024-04-27T01:09:53.979601+0000","last_active":"2024-04-27T01:10:31.028627+0000","last_peered":"2024-04-27T01:10:31.028627+0000","last_clean":"2024-04-27T01:10:31.028627+0000","last_became_active":"2024-04-27T01:09:53.979321+0000","last_became_peered":"2024-04-27T01:09:53.979321+0000","last_unstale":"2024-04-27T01:10:31.028627+0000","last_undegraded":"2024-04-27T01:10:31.028627+0000","last_fullsized":"2024-04-27T01:10:31.028627+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:15:38.136474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193198+0000","last_change":"2024-04-27T01:09:57.998811+0000","last_active":"2024-04-27T01:10:31.193198+0000","last_peered":"2024-04-27T01:10:31.193198+0000","last_clean":"2024-04-27T01:10:31.193198+0000","last_became_active":"2024-04-27T01:09:57.998508+0000","last_became_peered":"2024-04-27T01:09:57.998508+0000","last_unstale":"2024-04-27T01:10:31.193198+0000","last_undegraded":"2024-04-27T01:10:31.193198+0000","last_fullsized":"2024-04-27T01:10:31.193198+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:59:17.095866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193996+0000","last_change":"2024-04-27T01:09:55.989483+0000","last_active":"2024-04-27T01:10:31.193996+0000","last_peered":"2024-04-27T01:10:31.193996+0000","last_clean":"2024-04-27T01:10:31.193996+0000","last_became_active":"2024-04-27T01:09:55.989212+0000","last_became_peered":"2024-04-27T01:09:55.989212+0000","last_unstale":"2024-04-27T01:10:31.193996+0000","last_undegraded":"2024-04-27T01:10:31.193996+0000","last_fullsized":"2024-04-27T01:10:31.193996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:10.247737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193655+0000","last_change":"2024-04-27T01:09:51.976098+0000","last_active":"2024-04-27T01:10:31.193655+0000","last_peered":"2024-04-27T01:10:31.193655+0000","last_clean":"2024-04-27T01:10:31.193655+0000","last_became_active":"2024-04-27T01:09:51.975334+0000","last_became_peered":"2024-04-27T01:09:51.975334+0000","last_unstale":"2024-04-27T01:10:31.193655+0000","last_undegraded":"2024-04-27T01:10:31.193655+0000","last_fullsized":"2024-04-27T01:10:31.193655+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:19.106542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'11","reported_seq":51,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028901+0000","last_change":"2024-04-27T01:09:53.985313+0000","last_active":"2024-04-27T01:10:31.028901+0000","last_peered":"2024-04-27T01:10:31.028901+0000","last_clean":"2024-04-27T01:10:31.028901+0000","last_became_active":"2024-04-27T01:09:53.984991+0000","last_became_peered":"2024-04-27T01:09:53.984991+0000","last_unstale":"2024-04-27T01:10:31.028901+0000","last_undegraded":"2024-04-27T01:10:31.028901+0000","last_fullsized":"2024-04-27T01:10:31.028901+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:42:01.646439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029284+0000","last_change":"2024-04-27T01:09:58.490814+0000","last_active":"2024-04-27T01:10:31.029284+0000","last_peered":"2024-04-27T01:10:31.029284+0000","last_clean":"2024-04-27T01:10:31.029284+0000","last_became_active":"2024-04-27T01:09:58.490510+0000","last_became_peered":"2024-04-27T01:09:58.490510+0000","last_unstale":"2024-04-27T01:10:31.029284+0000","last_undegraded":"2024-04-27T01:10:31.029284+0000","last_fullsized":"2024-04-27T01:10:31.029284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:47:00.605910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.034852+0000","last_change":"2024-04-27T01:09:55.994105+0000","last_active":"2024-04-27T01:10:31.034852+0000","last_peered":"2024-04-27T01:10:31.034852+0000","last_clean":"2024-04-27T01:10:31.034852+0000","last_became_active":"2024-04-27T01:09:55.993581+0000","last_became_peered":"2024-04-27T01:09:55.993581+0000","last_unstale":"2024-04-27T01:10:31.034852+0000","last_undegraded":"2024-04-27T01:10:31.034852+0000","last_fullsized":"2024-04-27T01:10:31.034852+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:57:12.005476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.208665+0000","last_change":"2024-04-27T01:09:51.976414+0000","last_active":"2024-04-27T01:10:31.208665+0000","last_peered":"2024-04-27T01:10:31.208665+0000","last_clean":"2024-04-27T01:10:31.208665+0000","last_became_active":"2024-04-27T01:09:51.976154+0000","last_became_peered":"2024-04-27T01:09:51.976154+0000","last_unstale":"2024-04-27T01:10:31.208665+0000","last_undegraded":"2024-04-27T01:10:31.208665+0000","last_fullsized":"2024-04-27T01:10:31.208665+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:20.334383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.030298+0000","last_change":"2024-04-27T01:09:53.980941+0000","last_active":"2024-04-27T01:10:31.030298+0000","last_peered":"2024-04-27T01:10:31.030298+0000","last_clean":"2024-04-27T01:10:31.030298+0000","last_became_active":"2024-04-27T01:09:53.980600+0000","last_became_peered":"2024-04-27T01:09:53.980600+0000","last_unstale":"2024-04-27T01:10:31.030298+0000","last_undegraded":"2024-04-27T01:10:31.030298+0000","last_fullsized":"2024-04-27T01:10:31.030298+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:38:47.989649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.203924+0000","last_change":"2024-04-27T01:09:58.492176+0000","last_active":"2024-04-27T01:10:31.203924+0000","last_peered":"2024-04-27T01:10:31.203924+0000","last_clean":"2024-04-27T01:10:31.203924+0000","last_became_active":"2024-04-27T01:09:58.491717+0000","last_became_peered":"2024-04-27T01:09:58.491717+0000","last_unstale":"2024-04-27T01:10:31.203924+0000","last_undegraded":"2024-04-27T01:10:31.203924+0000","last_fullsized":"2024-04-27T01:10:31.203924+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:06:09.949561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028765+0000","last_change":"2024-04-27T01:09:55.992349+0000","last_active":"2024-04-27T01:10:31.028765+0000","last_peered":"2024-04-27T01:10:31.028765+0000","last_clean":"2024-04-27T01:10:31.028765+0000","last_became_active":"2024-04-27T01:09:55.992068+0000","last_became_peered":"2024-04-27T01:09:55.992068+0000","last_unstale":"2024-04-27T01:10:31.028765+0000","last_undegraded":"2024-04-27T01:10:31.028765+0000","last_fullsized":"2024-04-27T01:10:31.028765+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:42:15.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028648+0000","last_change":"2024-04-27T01:09:51.978740+0000","last_active":"2024-04-27T01:10:31.028648+0000","last_peered":"2024-04-27T01:10:31.028648+0000","last_clean":"2024-04-27T01:10:31.028648+0000","last_became_active":"2024-04-27T01:09:51.978419+0000","last_became_peered":"2024-04-27T01:09:51.978419+0000","last_unstale":"2024-04-27T01:10:31.028648+0000","last_undegraded":"2024-04-27T01:10:31.028648+0000","last_fullsized":"2024-04-27T01:10:31.028648+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:23:51.854276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'18","reported_seq":64,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.201801+0000","last_change":"2024-04-27T01:09:53.985482+0000","last_active":"2024-04-27T01:10:31.201801+0000","last_peered":"2024-04-27T01:10:31.201801+0000","last_clean":"2024-04-27T01:10:31.201801+0000","last_became_active":"2024-04-27T01:09:53.985099+0000","last_became_peered":"2024-04-27T01:09:53.985099+0000","last_unstale":"2024-04-27T01:10:31.201801+0000","last_undegraded":"2024-04-27T01:10:31.201801+0000","last_fullsized":"2024-04-27T01:10:31.201801+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:31:13.943321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195037+0000","last_change":"2024-04-27T01:09:57.996723+0000","last_active":"2024-04-27T01:10:31.195037+0000","last_peered":"2024-04-27T01:10:31.195037+0000","last_clean":"2024-04-27T01:10:31.195037+0000","last_became_active":"2024-04-27T01:09:57.995767+0000","last_became_peered":"2024-04-27T01:09:57.995767+0000","last_unstale":"2024-04-27T01:10:31.195037+0000","last_undegraded":"2024-04-27T01:10:31.195037+0000","last_fullsized":"2024-04-27T01:10:31.195037+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:00.665929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"69'8","reported_seq":53,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:07.041500+0000","last_change":"2024-04-27T01:09:55.987377+0000","last_active":"2024-04-27T01:11:07.041500+0000","last_peered":"2024-04-27T01:11:07.041500+0000","last_clean":"2024-04-27T01:11:07.041500+0000","last_became_active":"2024-04-27T01:09:55.987091+0000","last_became_peered":"2024-04-27T01:09:55.987091+0000","last_unstale":"2024-04-27T01:11:07.041500+0000","last_undegraded":"2024-04-27T01:11:07.041500+0000","last_fullsized":"2024-04-27T01:11:07.041500+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:10:08.810589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.033500+0000","last_change":"2024-04-27T01:09:51.975349+0000","last_active":"2024-04-27T01:10:31.033500+0000","last_peered":"2024-04-27T01:10:31.033500+0000","last_clean":"2024-04-27T01:10:31.033500+0000","last_became_active":"2024-04-27T01:09:51.975076+0000","last_became_peered":"2024-04-27T01:09:51.975076+0000","last_unstale":"2024-04-27T01:10:31.033500+0000","last_undegraded":"2024-04-27T01:10:31.033500+0000","last_fullsized":"2024-04-27T01:10:31.033500+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:52.747264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'16","reported_seq":56,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.212917+0000","last_change":"2024-04-27T01:09:54.097378+0000","last_active":"2024-04-27T01:10:31.212917+0000","last_peered":"2024-04-27T01:10:31.212917+0000","last_clean":"2024-04-27T01:10:31.212917+0000","last_became_active":"2024-04-27T01:09:54.097109+0000","last_became_peered":"2024-04-27T01:09:54.097109+0000","last_unstale":"2024-04-27T01:10:31.212917+0000","last_undegraded":"2024-04-27T01:10:31.212917+0000","last_fullsized":"2024-04-27T01:10:31.212917+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:29:15.858403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.034995+0000","last_change":"2024-04-27T01:09:57.995773+0000","last_active":"2024-04-27T01:10:31.034995+0000","last_peered":"2024-04-27T01:10:31.034995+0000","last_clean":"2024-04-27T01:10:31.034995+0000","last_became_active":"2024-04-27T01:09:57.995241+0000","last_became_peered":"2024-04-27T01:09:57.995241+0000","last_unstale":"2024-04-27T01:10:31.034995+0000","last_undegraded":"2024-04-27T01:10:31.034995+0000","last_fullsized":"2024-04-27T01:10:31.034995+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:54:52.336231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.213294+0000","last_change":"2024-04-27T01:09:55.983435+0000","last_active":"2024-04-27T01:10:31.213294+0000","last_peered":"2024-04-27T01:10:31.213294+0000","last_clean":"2024-04-27T01:10:31.213294+0000","last_became_active":"2024-04-27T01:09:55.983067+0000","last_became_peered":"2024-04-27T01:09:55.983067+0000","last_unstale":"2024-04-27T01:10:31.213294+0000","last_undegraded":"2024-04-27T01:10:31.213294+0000","last_fullsized":"2024-04-27T01:10:31.213294+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:49.791414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200589+0000","last_change":"2024-04-27T01:09:51.977196+0000","last_active":"2024-04-27T01:10:31.200589+0000","last_peered":"2024-04-27T01:10:31.200589+0000","last_clean":"2024-04-27T01:10:31.200589+0000","last_became_active":"2024-04-27T01:09:51.976906+0000","last_became_peered":"2024-04-27T01:09:51.976906+0000","last_unstale":"2024-04-27T01:10:31.200589+0000","last_undegraded":"2024-04-27T01:10:31.200589+0000","last_fullsized":"2024-04-27T01:10:31.200589+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:19:42.666108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":60,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195406+0000","last_change":"2024-04-27T01:09:53.982622+0000","last_active":"2024-04-27T01:10:31.195406+0000","last_peered":"2024-04-27T01:10:31.195406+0000","last_clean":"2024-04-27T01:10:31.195406+0000","last_became_active":"2024-04-27T01:09:53.982311+0000","last_became_peered":"2024-04-27T01:09:53.982311+0000","last_unstale":"2024-04-27T01:10:31.195406+0000","last_undegraded":"2024-04-27T01:10:31.195406+0000","last_fullsized":"2024-04-27T01:10:31.195406+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:05.219310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193899+0000","last_change":"2024-04-27T01:09:57.995625+0000","last_active":"2024-04-27T01:10:31.193899+0000","last_peered":"2024-04-27T01:10:31.193899+0000","last_clean":"2024-04-27T01:10:31.193899+0000","last_became_active":"2024-04-27T01:09:57.995033+0000","last_became_peered":"2024-04-27T01:09:57.995033+0000","last_unstale":"2024-04-27T01:10:31.193899+0000","last_undegraded":"2024-04-27T01:10:31.193899+0000","last_fullsized":"2024-04-27T01:10:31.193899+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:40:52.027696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193194+0000","last_change":"2024-04-27T01:09:55.986992+0000","last_active":"2024-04-27T01:10:31.193194+0000","last_peered":"2024-04-27T01:10:31.193194+0000","last_clean":"2024-04-27T01:10:31.193194+0000","last_became_active":"2024-04-27T01:09:55.986653+0000","last_became_peered":"2024-04-27T01:09:55.986653+0000","last_unstale":"2024-04-27T01:10:31.193194+0000","last_undegraded":"2024-04-27T01:10:31.193194+0000","last_fullsized":"2024-04-27T01:10:31.193194+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:48.395497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.214125+0000","last_change":"2024-04-27T01:09:51.980028+0000","last_active":"2024-04-27T01:10:31.214125+0000","last_peered":"2024-04-27T01:10:31.214125+0000","last_clean":"2024-04-27T01:10:31.214125+0000","last_became_active":"2024-04-27T01:09:51.979725+0000","last_became_peered":"2024-04-27T01:09:51.979725+0000","last_unstale":"2024-04-27T01:10:31.214125+0000","last_undegraded":"2024-04-27T01:10:31.214125+0000","last_fullsized":"2024-04-27T01:10:31.214125+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:03:16.016040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":35,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200685+0000","last_change":"2024-04-27T01:09:53.983929+0000","last_active":"2024-04-27T01:10:31.200685+0000","last_peered":"2024-04-27T01:10:31.200685+0000","last_clean":"2024-04-27T01:10:31.200685+0000","last_became_active":"2024-04-27T01:09:53.983592+0000","last_became_peered":"2024-04-27T01:09:53.983592+0000","last_unstale":"2024-04-27T01:10:31.200685+0000","last_undegraded":"2024-04-27T01:10:31.200685+0000","last_fullsized":"2024-04-27T01:10:31.200685+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:53.243435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200297+0000","last_change":"2024-04-27T01:09:58.492509+0000","last_active":"2024-04-27T01:10:31.200297+0000","last_peered":"2024-04-27T01:10:31.200297+0000","last_clean":"2024-04-27T01:10:31.200297+0000","last_became_active":"2024-04-27T01:09:58.492189+0000","last_became_peered":"2024-04-27T01:09:58.492189+0000","last_unstale":"2024-04-27T01:10:31.200297+0000","last_undegraded":"2024-04-27T01:10:31.200297+0000","last_fullsized":"2024-04-27T01:10:31.200297+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:41:44.975123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194815+0000","last_change":"2024-04-27T01:09:55.991580+0000","last_active":"2024-04-27T01:10:31.194815+0000","last_peered":"2024-04-27T01:10:31.194815+0000","last_clean":"2024-04-27T01:10:31.194815+0000","last_became_active":"2024-04-27T01:09:55.991287+0000","last_became_peered":"2024-04-27T01:09:55.991287+0000","last_unstale":"2024-04-27T01:10:31.194815+0000","last_undegraded":"2024-04-27T01:10:31.194815+0000","last_fullsized":"2024-04-27T01:10:31.194815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:42:08.400885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195058+0000","last_change":"2024-04-27T01:09:51.976544+0000","last_active":"2024-04-27T01:10:31.195058+0000","last_peered":"2024-04-27T01:10:31.195058+0000","last_clean":"2024-04-27T01:10:31.195058+0000","last_became_active":"2024-04-27T01:09:51.976196+0000","last_became_peered":"2024-04-27T01:09:51.976196+0000","last_unstale":"2024-04-27T01:10:31.195058+0000","last_undegraded":"2024-04-27T01:10:31.195058+0000","last_fullsized":"2024-04-27T01:10:31.195058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:53:45.288962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"69'17","reported_seq":60,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.204263+0000","last_change":"2024-04-27T01:09:54.096077+0000","last_active":"2024-04-27T01:10:31.204263+0000","last_peered":"2024-04-27T01:10:31.204263+0000","last_clean":"2024-04-27T01:10:31.204263+0000","last_became_active":"2024-04-27T01:09:54.095717+0000","last_became_peered":"2024-04-27T01:09:54.095717+0000","last_unstale":"2024-04-27T01:10:31.204263+0000","last_undegraded":"2024-04-27T01:10:31.204263+0000","last_fullsized":"2024-04-27T01:10:31.204263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:13:36.573405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202175+0000","last_change":"2024-04-27T01:09:58.003824+0000","last_active":"2024-04-27T01:10:31.202175+0000","last_peered":"2024-04-27T01:10:31.202175+0000","last_clean":"2024-04-27T01:10:31.202175+0000","last_became_active":"2024-04-27T01:09:58.003356+0000","last_became_peered":"2024-04-27T01:09:58.003356+0000","last_unstale":"2024-04-27T01:10:31.202175+0000","last_undegraded":"2024-04-27T01:10:31.202175+0000","last_fullsized":"2024-04-27T01:10:31.202175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:04.144673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202185+0000","last_change":"2024-04-27T01:09:55.995358+0000","last_active":"2024-04-27T01:10:31.202185+0000","last_peered":"2024-04-27T01:10:31.202185+0000","last_clean":"2024-04-27T01:10:31.202185+0000","last_became_active":"2024-04-27T01:09:55.994787+0000","last_became_peered":"2024-04-27T01:09:55.994787+0000","last_unstale":"2024-04-27T01:10:31.202185+0000","last_undegraded":"2024-04-27T01:10:31.202185+0000","last_fullsized":"2024-04-27T01:10:31.202185+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:43:08.388998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194645+0000","last_change":"2024-04-27T01:09:51.981304+0000","last_active":"2024-04-27T01:10:31.194645+0000","last_peered":"2024-04-27T01:10:31.194645+0000","last_clean":"2024-04-27T01:10:31.194645+0000","last_became_active":"2024-04-27T01:09:51.980959+0000","last_became_peered":"2024-04-27T01:09:51.980959+0000","last_unstale":"2024-04-27T01:10:31.194645+0000","last_undegraded":"2024-04-27T01:10:31.194645+0000","last_fullsized":"2024-04-27T01:10:31.194645+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:34:55.918660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.034264+0000","last_change":"2024-04-27T01:09:53.984436+0000","last_active":"2024-04-27T01:10:31.034264+0000","last_peered":"2024-04-27T01:10:31.034264+0000","last_clean":"2024-04-27T01:10:31.034264+0000","last_became_active":"2024-04-27T01:09:53.984121+0000","last_became_peered":"2024-04-27T01:09:53.984121+0000","last_unstale":"2024-04-27T01:10:31.034264+0000","last_undegraded":"2024-04-27T01:10:31.034264+0000","last_fullsized":"2024-04-27T01:10:31.034264+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:25.560995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.204159+0000","last_change":"2024-04-27T01:09:58.000375+0000","last_active":"2024-04-27T01:10:31.204159+0000","last_peered":"2024-04-27T01:10:31.204159+0000","last_clean":"2024-04-27T01:10:31.204159+0000","last_became_active":"2024-04-27T01:09:57.999943+0000","last_became_peered":"2024-04-27T01:09:57.999943+0000","last_unstale":"2024-04-27T01:10:31.204159+0000","last_undegraded":"2024-04-27T01:10:31.204159+0000","last_fullsized":"2024-04-27T01:10:31.204159+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:03.658855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.034036+0000","last_change":"2024-04-27T01:09:55.993241+0000","last_active":"2024-04-27T01:10:31.034036+0000","last_peered":"2024-04-27T01:10:31.034036+0000","last_clean":"2024-04-27T01:10:31.034036+0000","last_became_active":"2024-04-27T01:09:55.992901+0000","last_became_peered":"2024-04-27T01:09:55.992901+0000","last_unstale":"2024-04-27T01:10:31.034036+0000","last_undegraded":"2024-04-27T01:10:31.034036+0000","last_fullsized":"2024-04-27T01:10:31.034036+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:21:15.285498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028964+0000","last_change":"2024-04-27T01:09:51.979581+0000","last_active":"2024-04-27T01:10:31.028964+0000","last_peered":"2024-04-27T01:10:31.028964+0000","last_clean":"2024-04-27T01:10:31.028964+0000","last_became_active":"2024-04-27T01:09:51.979251+0000","last_became_peered":"2024-04-27T01:09:51.979251+0000","last_unstale":"2024-04-27T01:10:31.028964+0000","last_undegraded":"2024-04-27T01:10:31.028964+0000","last_fullsized":"2024-04-27T01:10:31.028964+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:04:15.479829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":66,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029226+0000","last_change":"2024-04-27T01:10:22.102174+0000","last_active":"2024-04-27T01:10:31.029226+0000","last_peered":"2024-04-27T01:10:31.029226+0000","last_clean":"2024-04-27T01:10:31.029226+0000","last_became_active":"2024-04-27T01:10:22.084998+0000","last_became_peered":"2024-04-27T01:10:22.084998+0000","last_unstale":"2024-04-27T01:10:31.029226+0000","last_undegraded":"2024-04-27T01:10:31.029226+0000","last_fullsized":"2024-04-27T01:10:31.029226+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:31.738300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.207946+0000","last_change":"2024-04-27T01:09:58.003556+0000","last_active":"2024-04-27T01:10:31.207946+0000","last_peered":"2024-04-27T01:10:31.207946+0000","last_clean":"2024-04-27T01:10:31.207946+0000","last_became_active":"2024-04-27T01:09:58.003224+0000","last_became_peered":"2024-04-27T01:09:58.003224+0000","last_unstale":"2024-04-27T01:10:31.207946+0000","last_undegraded":"2024-04-27T01:10:31.207946+0000","last_fullsized":"2024-04-27T01:10:31.207946+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:46:55.737761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:07.041522+0000","last_change":"2024-04-27T01:09:55.990672+0000","last_active":"2024-04-27T01:11:07.041522+0000","last_peered":"2024-04-27T01:11:07.041522+0000","last_clean":"2024-04-27T01:11:07.041522+0000","last_became_active":"2024-04-27T01:09:55.989696+0000","last_became_peered":"2024-04-27T01:09:55.989696+0000","last_unstale":"2024-04-27T01:11:07.041522+0000","last_undegraded":"2024-04-27T01:11:07.041522+0000","last_fullsized":"2024-04-27T01:11:07.041522+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:09.173488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193344+0000","last_change":"2024-04-27T01:09:51.978172+0000","last_active":"2024-04-27T01:10:31.193344+0000","last_peered":"2024-04-27T01:10:31.193344+0000","last_clean":"2024-04-27T01:10:31.193344+0000","last_became_active":"2024-04-27T01:09:51.977820+0000","last_became_peered":"2024-04-27T01:09:51.977820+0000","last_unstale":"2024-04-27T01:10:31.193344+0000","last_undegraded":"2024-04-27T01:10:31.193344+0000","last_fullsized":"2024-04-27T01:10:31.193344+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:58.795962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'23","reported_seq":69,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.201070+0000","last_change":"2024-04-27T01:09:54.096114+0000","last_active":"2024-04-27T01:10:31.201070+0000","last_peered":"2024-04-27T01:10:31.201070+0000","last_clean":"2024-04-27T01:10:31.201070+0000","last_became_active":"2024-04-27T01:09:54.095824+0000","last_became_peered":"2024-04-27T01:09:54.095824+0000","last_unstale":"2024-04-27T01:10:31.201070+0000","last_undegraded":"2024-04-27T01:10:31.201070+0000","last_fullsized":"2024-04-27T01:10:31.201070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:37.335086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029748+0000","last_change":"2024-04-27T01:09:57.996025+0000","last_active":"2024-04-27T01:10:31.029748+0000","last_peered":"2024-04-27T01:10:31.029748+0000","last_clean":"2024-04-27T01:10:31.029748+0000","last_became_active":"2024-04-27T01:09:57.995746+0000","last_became_peered":"2024-04-27T01:09:57.995746+0000","last_unstale":"2024-04-27T01:10:31.029748+0000","last_undegraded":"2024-04-27T01:10:31.029748+0000","last_fullsized":"2024-04-27T01:10:31.029748+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:23:54.706381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:07.041607+0000","last_change":"2024-04-27T01:09:55.990395+0000","last_active":"2024-04-27T01:11:07.041607+0000","last_peered":"2024-04-27T01:11:07.041607+0000","last_clean":"2024-04-27T01:11:07.041607+0000","last_became_active":"2024-04-27T01:09:55.990086+0000","last_became_peered":"2024-04-27T01:09:55.990086+0000","last_unstale":"2024-04-27T01:11:07.041607+0000","last_undegraded":"2024-04-27T01:11:07.041607+0000","last_fullsized":"2024-04-27T01:11:07.041607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:03.299934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.209784+0000","last_change":"2024-04-27T01:09:51.976764+0000","last_active":"2024-04-27T01:10:31.209784+0000","last_peered":"2024-04-27T01:10:31.209784+0000","last_clean":"2024-04-27T01:10:31.209784+0000","last_became_active":"2024-04-27T01:09:51.976161+0000","last_became_peered":"2024-04-27T01:09:51.976161+0000","last_unstale":"2024-04-27T01:10:31.209784+0000","last_undegraded":"2024-04-27T01:10:31.209784+0000","last_fullsized":"2024-04-27T01:10:31.209784+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:06:04.577915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'15","reported_seq":57,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194507+0000","last_change":"2024-04-27T01:09:54.097924+0000","last_active":"2024-04-27T01:10:31.194507+0000","last_peered":"2024-04-27T01:10:31.194507+0000","last_clean":"2024-04-27T01:10:31.194507+0000","last_became_active":"2024-04-27T01:09:54.097583+0000","last_became_peered":"2024-04-27T01:09:54.097583+0000","last_unstale":"2024-04-27T01:10:31.194507+0000","last_undegraded":"2024-04-27T01:10:31.194507+0000","last_fullsized":"2024-04-27T01:10:31.194507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:12:03.954912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.030690+0000","last_change":"2024-04-27T01:09:57.993895+0000","last_active":"2024-04-27T01:10:31.030690+0000","last_peered":"2024-04-27T01:10:31.030690+0000","last_clean":"2024-04-27T01:10:31.030690+0000","last_became_active":"2024-04-27T01:09:57.993564+0000","last_became_peered":"2024-04-27T01:09:57.993564+0000","last_unstale":"2024-04-27T01:10:31.030690+0000","last_undegraded":"2024-04-27T01:10:31.030690+0000","last_fullsized":"2024-04-27T01:10:31.030690+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:12:41.054063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.203834+0000","last_change":"2024-04-27T01:09:55.991784+0000","last_active":"2024-04-27T01:10:31.203834+0000","last_peered":"2024-04-27T01:10:31.203834+0000","last_clean":"2024-04-27T01:10:31.203834+0000","last_became_active":"2024-04-27T01:09:55.991441+0000","last_became_peered":"2024-04-27T01:09:55.991441+0000","last_unstale":"2024-04-27T01:10:31.203834+0000","last_undegraded":"2024-04-27T01:10:31.203834+0000","last_fullsized":"2024-04-27T01:10:31.203834+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:34.613206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193549+0000","last_change":"2024-04-27T01:09:51.977480+0000","last_active":"2024-04-27T01:10:31.193549+0000","last_peered":"2024-04-27T01:10:31.193549+0000","last_clean":"2024-04-27T01:10:31.193549+0000","last_became_active":"2024-04-27T01:09:51.977157+0000","last_became_peered":"2024-04-27T01:09:51.977157+0000","last_unstale":"2024-04-27T01:10:31.193549+0000","last_undegraded":"2024-04-27T01:10:31.193549+0000","last_fullsized":"2024-04-27T01:10:31.193549+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:54.830982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'21","reported_seq":69,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.213088+0000","last_change":"2024-04-27T01:09:54.097131+0000","last_active":"2024-04-27T01:10:31.213088+0000","last_peered":"2024-04-27T01:10:31.213088+0000","last_clean":"2024-04-27T01:10:31.213088+0000","last_became_active":"2024-04-27T01:09:54.096854+0000","last_became_peered":"2024-04-27T01:09:54.096854+0000","last_unstale":"2024-04-27T01:10:31.213088+0000","last_undegraded":"2024-04-27T01:10:31.213088+0000","last_fullsized":"2024-04-27T01:10:31.213088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:45:27.541174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194550+0000","last_change":"2024-04-27T01:09:58.492699+0000","last_active":"2024-04-27T01:10:31.194550+0000","last_peered":"2024-04-27T01:10:31.194550+0000","last_clean":"2024-04-27T01:10:31.194550+0000","last_became_active":"2024-04-27T01:09:58.492357+0000","last_became_peered":"2024-04-27T01:09:58.492357+0000","last_unstale":"2024-04-27T01:10:31.194550+0000","last_undegraded":"2024-04-27T01:10:31.194550+0000","last_fullsized":"2024-04-27T01:10:31.194550+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:17:39.800109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195146+0000","last_change":"2024-04-27T01:09:55.990121+0000","last_active":"2024-04-27T01:10:31.195146+0000","last_peered":"2024-04-27T01:10:31.195146+0000","last_clean":"2024-04-27T01:10:31.195146+0000","last_became_active":"2024-04-27T01:09:55.989627+0000","last_became_peered":"2024-04-27T01:09:55.989627+0000","last_unstale":"2024-04-27T01:10:31.195146+0000","last_undegraded":"2024-04-27T01:10:31.195146+0000","last_fullsized":"2024-04-27T01:10:31.195146+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:37:17.381011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193879+0000","last_change":"2024-04-27T01:09:51.975778+0000","last_active":"2024-04-27T01:10:31.193879+0000","last_peered":"2024-04-27T01:10:31.193879+0000","last_clean":"2024-04-27T01:10:31.193879+0000","last_became_active":"2024-04-27T01:09:51.975469+0000","last_became_peered":"2024-04-27T01:09:51.975469+0000","last_unstale":"2024-04-27T01:10:31.193879+0000","last_undegraded":"2024-04-27T01:10:31.193879+0000","last_fullsized":"2024-04-27T01:10:31.193879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:46:26.134298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'12","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.204024+0000","last_change":"2024-04-27T01:09:54.096271+0000","last_active":"2024-04-27T01:10:31.204024+0000","last_peered":"2024-04-27T01:10:31.204024+0000","last_clean":"2024-04-27T01:10:31.204024+0000","last_became_active":"2024-04-27T01:09:54.095897+0000","last_became_peered":"2024-04-27T01:09:54.095897+0000","last_unstale":"2024-04-27T01:10:31.204024+0000","last_undegraded":"2024-04-27T01:10:31.204024+0000","last_fullsized":"2024-04-27T01:10:31.204024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:54.537542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193843+0000","last_change":"2024-04-27T01:09:58.492112+0000","last_active":"2024-04-27T01:10:31.193843+0000","last_peered":"2024-04-27T01:10:31.193843+0000","last_clean":"2024-04-27T01:10:31.193843+0000","last_became_active":"2024-04-27T01:09:58.491814+0000","last_became_peered":"2024-04-27T01:09:58.491814+0000","last_unstale":"2024-04-27T01:10:31.193843+0000","last_undegraded":"2024-04-27T01:10:31.193843+0000","last_fullsized":"2024-04-27T01:10:31.193843+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:12.127109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":45,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195152+0000","last_change":"2024-04-27T01:09:51.984095+0000","last_active":"2024-04-27T01:10:31.195152+0000","last_peered":"2024-04-27T01:10:31.195152+0000","last_clean":"2024-04-27T01:10:31.195152+0000","last_became_active":"2024-04-27T01:09:51.983527+0000","last_became_peered":"2024-04-27T01:09:51.983527+0000","last_unstale":"2024-04-27T01:10:31.195152+0000","last_undegraded":"2024-04-27T01:10:31.195152+0000","last_fullsized":"2024-04-27T01:10:31.195152+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:07.781337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"69'28","reported_seq":79,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193074+0000","last_change":"2024-04-27T01:09:54.098741+0000","last_active":"2024-04-27T01:10:31.193074+0000","last_peered":"2024-04-27T01:10:31.193074+0000","last_clean":"2024-04-27T01:10:31.193074+0000","last_became_active":"2024-04-27T01:09:54.098422+0000","last_became_peered":"2024-04-27T01:09:54.098422+0000","last_unstale":"2024-04-27T01:10:31.193074+0000","last_undegraded":"2024-04-27T01:10:31.193074+0000","last_fullsized":"2024-04-27T01:10:31.193074+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:27.798531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:07.041512+0000","last_change":"2024-04-27T01:09:55.992444+0000","last_active":"2024-04-27T01:11:07.041512+0000","last_peered":"2024-04-27T01:11:07.041512+0000","last_clean":"2024-04-27T01:11:07.041512+0000","last_became_active":"2024-04-27T01:09:55.991936+0000","last_became_peered":"2024-04-27T01:09:55.991936+0000","last_unstale":"2024-04-27T01:11:07.041512+0000","last_undegraded":"2024-04-27T01:11:07.041512+0000","last_fullsized":"2024-04-27T01:11:07.041512+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:22:04.321113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200404+0000","last_change":"2024-04-27T01:09:58.000147+0000","last_active":"2024-04-27T01:10:31.200404+0000","last_peered":"2024-04-27T01:10:31.200404+0000","last_clean":"2024-04-27T01:10:31.200404+0000","last_became_active":"2024-04-27T01:09:57.999476+0000","last_became_peered":"2024-04-27T01:09:57.999476+0000","last_unstale":"2024-04-27T01:10:31.200404+0000","last_undegraded":"2024-04-27T01:10:31.200404+0000","last_fullsized":"2024-04-27T01:10:31.200404+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:07.164832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194030+0000","last_change":"2024-04-27T01:09:51.981342+0000","last_active":"2024-04-27T01:10:31.194030+0000","last_peered":"2024-04-27T01:10:31.194030+0000","last_clean":"2024-04-27T01:10:31.194030+0000","last_became_active":"2024-04-27T01:09:51.981012+0000","last_became_peered":"2024-04-27T01:09:51.981012+0000","last_unstale":"2024-04-27T01:10:31.194030+0000","last_undegraded":"2024-04-27T01:10:31.194030+0000","last_fullsized":"2024-04-27T01:10:31.194030+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:39.300351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'24","reported_seq":76,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029374+0000","last_change":"2024-04-27T01:09:53.980354+0000","last_active":"2024-04-27T01:10:31.029374+0000","last_peered":"2024-04-27T01:10:31.029374+0000","last_clean":"2024-04-27T01:10:31.029374+0000","last_became_active":"2024-04-27T01:09:53.979993+0000","last_became_peered":"2024-04-27T01:09:53.979993+0000","last_unstale":"2024-04-27T01:10:31.029374+0000","last_undegraded":"2024-04-27T01:10:31.029374+0000","last_fullsized":"2024-04-27T01:10:31.029374+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:47.872365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029326+0000","last_change":"2024-04-27T01:09:55.987622+0000","last_active":"2024-04-27T01:10:31.029326+0000","last_peered":"2024-04-27T01:10:31.029326+0000","last_clean":"2024-04-27T01:10:31.029326+0000","last_became_active":"2024-04-27T01:09:55.987286+0000","last_became_peered":"2024-04-27T01:09:55.987286+0000","last_unstale":"2024-04-27T01:10:31.029326+0000","last_undegraded":"2024-04-27T01:10:31.029326+0000","last_fullsized":"2024-04-27T01:10:31.029326+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:59:58.984779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"69'1","reported_seq":22,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194293+0000","last_change":"2024-04-27T01:09:57.997122+0000","last_active":"2024-04-27T01:10:31.194293+0000","last_peered":"2024-04-27T01:10:31.194293+0000","last_clean":"2024-04-27T01:10:31.194293+0000","last_became_active":"2024-04-27T01:09:57.996499+0000","last_became_peered":"2024-04-27T01:09:57.996499+0000","last_unstale":"2024-04-27T01:10:31.194293+0000","last_undegraded":"2024-04-27T01:10:31.194293+0000","last_fullsized":"2024-04-27T01:10:31.194293+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:50:31.513681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.030144+0000","last_change":"2024-04-27T01:09:55.988725+0000","last_active":"2024-04-27T01:10:31.030144+0000","last_peered":"2024-04-27T01:10:31.030144+0000","last_clean":"2024-04-27T01:10:31.030144+0000","last_became_active":"2024-04-27T01:09:55.988394+0000","last_became_peered":"2024-04-27T01:09:55.988394+0000","last_unstale":"2024-04-27T01:10:31.030144+0000","last_undegraded":"2024-04-27T01:10:31.030144+0000","last_fullsized":"2024-04-27T01:10:31.030144+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:17:19.998896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194879+0000","last_change":"2024-04-27T01:09:51.979406+0000","last_active":"2024-04-27T01:10:31.194879+0000","last_peered":"2024-04-27T01:10:31.194879+0000","last_clean":"2024-04-27T01:10:31.194879+0000","last_became_active":"2024-04-27T01:09:51.979078+0000","last_became_peered":"2024-04-27T01:09:51.979078+0000","last_unstale":"2024-04-27T01:10:31.194879+0000","last_undegraded":"2024-04-27T01:10:31.194879+0000","last_fullsized":"2024-04-27T01:10:31.194879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:06:18.209863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'25","reported_seq":72,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194402+0000","last_change":"2024-04-27T01:09:54.096387+0000","last_active":"2024-04-27T01:10:31.194402+0000","last_peered":"2024-04-27T01:10:31.194402+0000","last_clean":"2024-04-27T01:10:31.194402+0000","last_became_active":"2024-04-27T01:09:54.096056+0000","last_became_peered":"2024-04-27T01:09:54.096056+0000","last_unstale":"2024-04-27T01:10:31.194402+0000","last_undegraded":"2024-04-27T01:10:31.194402+0000","last_fullsized":"2024-04-27T01:10:31.194402+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:12:17.119595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202303+0000","last_change":"2024-04-27T01:09:58.000860+0000","last_active":"2024-04-27T01:10:31.202303+0000","last_peered":"2024-04-27T01:10:31.202303+0000","last_clean":"2024-04-27T01:10:31.202303+0000","last_became_active":"2024-04-27T01:09:58.000552+0000","last_became_peered":"2024-04-27T01:09:58.000552+0000","last_unstale":"2024-04-27T01:10:31.202303+0000","last_undegraded":"2024-04-27T01:10:31.202303+0000","last_fullsized":"2024-04-27T01:10:31.202303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:38.802150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029932+0000","last_change":"2024-04-27T01:09:55.986655+0000","last_active":"2024-04-27T01:10:31.029932+0000","last_peered":"2024-04-27T01:10:31.029932+0000","last_clean":"2024-04-27T01:10:31.029932+0000","last_became_active":"2024-04-27T01:09:55.986294+0000","last_became_peered":"2024-04-27T01:09:55.986294+0000","last_unstale":"2024-04-27T01:10:31.029932+0000","last_undegraded":"2024-04-27T01:10:31.029932+0000","last_fullsized":"2024-04-27T01:10:31.029932+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:07:26.356079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193752+0000","last_change":"2024-04-27T01:09:51.974394+0000","last_active":"2024-04-27T01:10:31.193752+0000","last_peered":"2024-04-27T01:10:31.193752+0000","last_clean":"2024-04-27T01:10:31.193752+0000","last_became_active":"2024-04-27T01:09:51.974100+0000","last_became_peered":"2024-04-27T01:09:51.974100+0000","last_unstale":"2024-04-27T01:10:31.193752+0000","last_undegraded":"2024-04-27T01:10:31.193752+0000","last_fullsized":"2024-04-27T01:10:31.193752+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:26.036970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"69'15","reported_seq":57,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.208967+0000","last_change":"2024-04-27T01:09:53.985351+0000","last_active":"2024-04-27T01:10:31.208967+0000","last_peered":"2024-04-27T01:10:31.208967+0000","last_clean":"2024-04-27T01:10:31.208967+0000","last_became_active":"2024-04-27T01:09:53.985000+0000","last_became_peered":"2024-04-27T01:09:53.985000+0000","last_unstale":"2024-04-27T01:10:31.208967+0000","last_undegraded":"2024-04-27T01:10:31.208967+0000","last_fullsized":"2024-04-27T01:10:31.208967+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:11.115107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202780+0000","last_change":"2024-04-27T01:09:57.996524+0000","last_active":"2024-04-27T01:10:31.202780+0000","last_peered":"2024-04-27T01:10:31.202780+0000","last_clean":"2024-04-27T01:10:31.202780+0000","last_became_active":"2024-04-27T01:09:57.996163+0000","last_became_peered":"2024-04-27T01:09:57.996163+0000","last_unstale":"2024-04-27T01:10:31.202780+0000","last_undegraded":"2024-04-27T01:10:31.202780+0000","last_fullsized":"2024-04-27T01:10:31.202780+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:50.034133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":44,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193538+0000","last_change":"2024-04-27T01:09:54.974415+0000","last_active":"2024-04-27T01:10:31.193538+0000","last_peered":"2024-04-27T01:10:31.193538+0000","last_clean":"2024-04-27T01:10:31.193538+0000","last_became_active":"2024-04-27T01:09:50.967576+0000","last_became_peered":"2024-04-27T01:09:50.967576+0000","last_unstale":"2024-04-27T01:10:31.193538+0000","last_undegraded":"2024-04-27T01:10:31.193538+0000","last_fullsized":"2024-04-27T01:10:31.193538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_clean_scrub_stamp":"2024-04-27T01:09:49.954065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:20.524736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030547220000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028778+0000","last_change":"2024-04-27T01:09:51.978554+0000","last_active":"2024-04-27T01:10:31.028778+0000","last_peered":"2024-04-27T01:10:31.028778+0000","last_clean":"2024-04-27T01:10:31.028778+0000","last_became_active":"2024-04-27T01:09:51.977736+0000","last_became_peered":"2024-04-27T01:09:51.977736+0000","last_unstale":"2024-04-27T01:10:31.028778+0000","last_undegraded":"2024-04-27T01:10:31.028778+0000","last_fullsized":"2024-04-27T01:10:31.028778+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:42.662246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"69'34","reported_seq":105,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040250+0000","last_change":"2024-04-27T01:09:53.984454+0000","last_active":"2024-04-27T01:11:02.040250+0000","last_peered":"2024-04-27T01:11:02.040250+0000","last_clean":"2024-04-27T01:11:02.040250+0000","last_became_active":"2024-04-27T01:09:53.983985+0000","last_became_peered":"2024-04-27T01:09:53.983985+0000","last_unstale":"2024-04-27T01:11:02.040250+0000","last_undegraded":"2024-04-27T01:11:02.040250+0000","last_fullsized":"2024-04-27T01:11:02.040250+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:13:37.096503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.208156+0000","last_change":"2024-04-27T01:09:55.990001+0000","last_active":"2024-04-27T01:10:31.208156+0000","last_peered":"2024-04-27T01:10:31.208156+0000","last_clean":"2024-04-27T01:10:31.208156+0000","last_became_active":"2024-04-27T01:09:55.989223+0000","last_became_peered":"2024-04-27T01:09:55.989223+0000","last_unstale":"2024-04-27T01:10:31.208156+0000","last_undegraded":"2024-04-27T01:10:31.208156+0000","last_fullsized":"2024-04-27T01:10:31.208156+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:20.167194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195036+0000","last_change":"2024-04-27T01:09:58.491893+0000","last_active":"2024-04-27T01:10:31.195036+0000","last_peered":"2024-04-27T01:10:31.195036+0000","last_clean":"2024-04-27T01:10:31.195036+0000","last_became_active":"2024-04-27T01:09:58.491600+0000","last_became_peered":"2024-04-27T01:09:58.491600+0000","last_unstale":"2024-04-27T01:10:31.195036+0000","last_undegraded":"2024-04-27T01:10:31.195036+0000","last_fullsized":"2024-04-27T01:10:31.195036+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:24:50.452967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.039268+0000","last_change":"2024-04-27T01:09:51.978284+0000","last_active":"2024-04-27T01:10:31.039268+0000","last_peered":"2024-04-27T01:10:31.039268+0000","last_clean":"2024-04-27T01:10:31.039268+0000","last_became_active":"2024-04-27T01:09:51.977919+0000","last_became_peered":"2024-04-27T01:09:51.977919+0000","last_unstale":"2024-04-27T01:10:31.039268+0000","last_undegraded":"2024-04-27T01:10:31.039268+0000","last_fullsized":"2024-04-27T01:10:31.039268+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:35:48.115350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029613+0000","last_change":"2024-04-27T01:09:54.971402+0000","last_active":"2024-04-27T01:10:31.029613+0000","last_peered":"2024-04-27T01:10:31.029613+0000","last_clean":"2024-04-27T01:10:31.029613+0000","last_became_active":"2024-04-27T01:09:50.970061+0000","last_became_peered":"2024-04-27T01:09:50.970061+0000","last_unstale":"2024-04-27T01:10:31.029613+0000","last_undegraded":"2024-04-27T01:10:31.029613+0000","last_fullsized":"2024-04-27T01:10:31.029613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_clean_scrub_stamp":"2024-04-27T01:09:49.954065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:22:48.313387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079872600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"69'19","reported_seq":68,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.038723+0000","last_change":"2024-04-27T01:09:53.984841+0000","last_active":"2024-04-27T01:10:31.038723+0000","last_peered":"2024-04-27T01:10:31.038723+0000","last_clean":"2024-04-27T01:10:31.038723+0000","last_became_active":"2024-04-27T01:09:53.984456+0000","last_became_peered":"2024-04-27T01:09:53.984456+0000","last_unstale":"2024-04-27T01:10:31.038723+0000","last_undegraded":"2024-04-27T01:10:31.038723+0000","last_fullsized":"2024-04-27T01:10:31.038723+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:30:15.182357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.030551+0000","last_change":"2024-04-27T01:09:55.989859+0000","last_active":"2024-04-27T01:10:31.030551+0000","last_peered":"2024-04-27T01:10:31.030551+0000","last_clean":"2024-04-27T01:10:31.030551+0000","last_became_active":"2024-04-27T01:09:55.989431+0000","last_became_peered":"2024-04-27T01:09:55.989431+0000","last_unstale":"2024-04-27T01:10:31.030551+0000","last_undegraded":"2024-04-27T01:10:31.030551+0000","last_fullsized":"2024-04-27T01:10:31.030551+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:45.939447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195032+0000","last_change":"2024-04-27T01:09:57.998807+0000","last_active":"2024-04-27T01:10:31.195032+0000","last_peered":"2024-04-27T01:10:31.195032+0000","last_clean":"2024-04-27T01:10:31.195032+0000","last_became_active":"2024-04-27T01:09:57.998457+0000","last_became_peered":"2024-04-27T01:09:57.998457+0000","last_unstale":"2024-04-27T01:10:31.195032+0000","last_undegraded":"2024-04-27T01:10:31.195032+0000","last_fullsized":"2024-04-27T01:10:31.195032+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:34:31.015288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'128","reported_seq":278,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:33.341976+0000","last_change":"2024-04-27T01:09:41.777614+0000","last_active":"2024-04-27T01:10:33.341976+0000","last_peered":"2024-04-27T01:10:33.341976+0000","last_clean":"2024-04-27T01:10:33.341976+0000","last_became_active":"2024-04-27T01:09:41.777074+0000","last_became_peered":"2024-04-27T01:09:41.777074+0000","last_unstale":"2024-04-27T01:10:33.341976+0000","last_undegraded":"2024-04-27T01:10:33.341976+0000","last_fullsized":"2024-04-27T01:10:33.341976+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:07:37.632381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:07:37.632381+0000","last_clean_scrub_stamp":"2024-04-27T01:07:37.632381+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:52.945958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200573+0000","last_change":"2024-04-27T01:09:51.978521+0000","last_active":"2024-04-27T01:10:31.200573+0000","last_peered":"2024-04-27T01:10:31.200573+0000","last_clean":"2024-04-27T01:10:31.200573+0000","last_became_active":"2024-04-27T01:09:51.978240+0000","last_became_peered":"2024-04-27T01:09:51.978240+0000","last_unstale":"2024-04-27T01:10:31.200573+0000","last_undegraded":"2024-04-27T01:10:31.200573+0000","last_fullsized":"2024-04-27T01:10:31.200573+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:42.911721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":68,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:07.041732+0000","last_change":"2024-04-27T01:09:54.096643+0000","last_active":"2024-04-27T01:11:07.041732+0000","last_peered":"2024-04-27T01:11:07.041732+0000","last_clean":"2024-04-27T01:11:07.041732+0000","last_became_active":"2024-04-27T01:09:54.096283+0000","last_became_peered":"2024-04-27T01:09:54.096283+0000","last_unstale":"2024-04-27T01:11:07.041732+0000","last_undegraded":"2024-04-27T01:11:07.041732+0000","last_fullsized":"2024-04-27T01:11:07.041732+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:15:47.195945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195207+0000","last_change":"2024-04-27T01:09:55.987753+0000","last_active":"2024-04-27T01:10:31.195207+0000","last_peered":"2024-04-27T01:10:31.195207+0000","last_clean":"2024-04-27T01:10:31.195207+0000","last_became_active":"2024-04-27T01:09:55.987000+0000","last_became_peered":"2024-04-27T01:09:55.987000+0000","last_unstale":"2024-04-27T01:10:31.195207+0000","last_undegraded":"2024-04-27T01:10:31.195207+0000","last_fullsized":"2024-04-27T01:10:31.195207+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:18.394267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.039496+0000","last_change":"2024-04-27T01:09:57.997995+0000","last_active":"2024-04-27T01:10:31.039496+0000","last_peered":"2024-04-27T01:10:31.039496+0000","last_clean":"2024-04-27T01:10:31.039496+0000","last_became_active":"2024-04-27T01:09:57.997664+0000","last_became_peered":"2024-04-27T01:09:57.997664+0000","last_unstale":"2024-04-27T01:10:31.039496+0000","last_undegraded":"2024-04-27T01:10:31.039496+0000","last_fullsized":"2024-04-27T01:10:31.039496+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:04.460907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"62'1","reported_seq":39,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202595+0000","last_change":"2024-04-27T01:09:51.975394+0000","last_active":"2024-04-27T01:10:31.202595+0000","last_peered":"2024-04-27T01:10:31.202595+0000","last_clean":"2024-04-27T01:10:31.202595+0000","last_became_active":"2024-04-27T01:09:51.974515+0000","last_became_peered":"2024-04-27T01:09:51.974515+0000","last_unstale":"2024-04-27T01:10:31.202595+0000","last_undegraded":"2024-04-27T01:10:31.202595+0000","last_fullsized":"2024-04-27T01:10:31.202595+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:28:50.723732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":117,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:10.366477+0000","last_change":"2024-04-27T01:09:54.974030+0000","last_active":"2024-04-27T01:11:10.366477+0000","last_peered":"2024-04-27T01:11:10.366477+0000","last_clean":"2024-04-27T01:11:10.366477+0000","last_became_active":"2024-04-27T01:09:50.967643+0000","last_became_peered":"2024-04-27T01:09:50.967643+0000","last_unstale":"2024-04-27T01:11:10.366477+0000","last_undegraded":"2024-04-27T01:11:10.366477+0000","last_fullsized":"2024-04-27T01:11:10.366477+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:49.954065+0000","last_clean_scrub_stamp":"2024-04-27T01:09:49.954065+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:02:23.146239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000441765,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":72,"num_read_kb":67,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":47,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.208084+0000","last_change":"2024-04-27T01:09:53.981423+0000","last_active":"2024-04-27T01:10:31.208084+0000","last_peered":"2024-04-27T01:10:31.208084+0000","last_clean":"2024-04-27T01:10:31.208084+0000","last_became_active":"2024-04-27T01:09:53.980996+0000","last_became_peered":"2024-04-27T01:09:53.980996+0000","last_unstale":"2024-04-27T01:10:31.208084+0000","last_undegraded":"2024-04-27T01:10:31.208084+0000","last_fullsized":"2024-04-27T01:10:31.208084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:37.319973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195444+0000","last_change":"2024-04-27T01:09:55.994152+0000","last_active":"2024-04-27T01:10:31.195444+0000","last_peered":"2024-04-27T01:10:31.195444+0000","last_clean":"2024-04-27T01:10:31.195444+0000","last_became_active":"2024-04-27T01:09:55.993872+0000","last_became_peered":"2024-04-27T01:09:55.993872+0000","last_unstale":"2024-04-27T01:10:31.195444+0000","last_undegraded":"2024-04-27T01:10:31.195444+0000","last_fullsized":"2024-04-27T01:10:31.195444+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:11:49.192838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.033302+0000","last_change":"2024-04-27T01:09:58.491368+0000","last_active":"2024-04-27T01:10:31.033302+0000","last_peered":"2024-04-27T01:10:31.033302+0000","last_clean":"2024-04-27T01:10:31.033302+0000","last_became_active":"2024-04-27T01:09:58.490925+0000","last_became_peered":"2024-04-27T01:09:58.490925+0000","last_unstale":"2024-04-27T01:10:31.033302+0000","last_undegraded":"2024-04-27T01:10:31.033302+0000","last_fullsized":"2024-04-27T01:10:31.033302+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:41:53.417660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.039810+0000","last_change":"2024-04-27T01:09:51.979283+0000","last_active":"2024-04-27T01:10:31.039810+0000","last_peered":"2024-04-27T01:10:31.039810+0000","last_clean":"2024-04-27T01:10:31.039810+0000","last_became_active":"2024-04-27T01:09:51.978963+0000","last_became_peered":"2024-04-27T01:09:51.978963+0000","last_unstale":"2024-04-27T01:10:31.039810+0000","last_undegraded":"2024-04-27T01:10:31.039810+0000","last_fullsized":"2024-04-27T01:10:31.039810+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:58:27.383762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"69'25","reported_seq":77,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.208764+0000","last_change":"2024-04-27T01:09:54.096966+0000","last_active":"2024-04-27T01:10:31.208764+0000","last_peered":"2024-04-27T01:10:31.208764+0000","last_clean":"2024-04-27T01:10:31.208764+0000","last_became_active":"2024-04-27T01:09:54.096116+0000","last_became_peered":"2024-04-27T01:09:54.096116+0000","last_unstale":"2024-04-27T01:10:31.208764+0000","last_undegraded":"2024-04-27T01:10:31.208764+0000","last_fullsized":"2024-04-27T01:10:31.208764+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:11:54.089352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194466+0000","last_change":"2024-04-27T01:09:55.985443+0000","last_active":"2024-04-27T01:10:31.194466+0000","last_peered":"2024-04-27T01:10:31.194466+0000","last_clean":"2024-04-27T01:10:31.194466+0000","last_became_active":"2024-04-27T01:09:55.984939+0000","last_became_peered":"2024-04-27T01:09:55.984939+0000","last_unstale":"2024-04-27T01:10:31.194466+0000","last_undegraded":"2024-04-27T01:10:31.194466+0000","last_fullsized":"2024-04-27T01:10:31.194466+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:31:25.407393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202216+0000","last_change":"2024-04-27T01:09:58.004290+0000","last_active":"2024-04-27T01:10:31.202216+0000","last_peered":"2024-04-27T01:10:31.202216+0000","last_clean":"2024-04-27T01:10:31.202216+0000","last_became_active":"2024-04-27T01:09:58.003641+0000","last_became_peered":"2024-04-27T01:09:58.003641+0000","last_unstale":"2024-04-27T01:10:31.202216+0000","last_undegraded":"2024-04-27T01:10:31.202216+0000","last_fullsized":"2024-04-27T01:10:31.202216+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:05.432205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"62'1","reported_seq":39,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194341+0000","last_change":"2024-04-27T01:09:51.975079+0000","last_active":"2024-04-27T01:10:31.194341+0000","last_peered":"2024-04-27T01:10:31.194341+0000","last_clean":"2024-04-27T01:10:31.194341+0000","last_became_active":"2024-04-27T01:09:51.974755+0000","last_became_peered":"2024-04-27T01:09:51.974755+0000","last_unstale":"2024-04-27T01:10:31.194341+0000","last_undegraded":"2024-04-27T01:10:31.194341+0000","last_fullsized":"2024-04-27T01:10:31.194341+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:23:00.685692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":56,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.034130+0000","last_change":"2024-04-27T01:09:53.982529+0000","last_active":"2024-04-27T01:10:31.034130+0000","last_peered":"2024-04-27T01:10:31.034130+0000","last_clean":"2024-04-27T01:10:31.034130+0000","last_became_active":"2024-04-27T01:09:53.982188+0000","last_became_peered":"2024-04-27T01:09:53.982188+0000","last_unstale":"2024-04-27T01:10:31.034130+0000","last_undegraded":"2024-04-27T01:10:31.034130+0000","last_fullsized":"2024-04-27T01:10:31.034130+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:52.168079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:07.041265+0000","last_change":"2024-04-27T01:09:55.986183+0000","last_active":"2024-04-27T01:11:07.041265+0000","last_peered":"2024-04-27T01:11:07.041265+0000","last_clean":"2024-04-27T01:11:07.041265+0000","last_became_active":"2024-04-27T01:09:55.985776+0000","last_became_peered":"2024-04-27T01:09:55.985776+0000","last_unstale":"2024-04-27T01:11:07.041265+0000","last_undegraded":"2024-04-27T01:11:07.041265+0000","last_fullsized":"2024-04-27T01:11:07.041265+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:13.714957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.193722+0000","last_change":"2024-04-27T01:09:58.493484+0000","last_active":"2024-04-27T01:10:31.193722+0000","last_peered":"2024-04-27T01:10:31.193722+0000","last_clean":"2024-04-27T01:10:31.193722+0000","last_became_active":"2024-04-27T01:09:58.493199+0000","last_became_peered":"2024-04-27T01:09:58.493199+0000","last_unstale":"2024-04-27T01:10:31.193722+0000","last_undegraded":"2024-04-27T01:10:31.193722+0000","last_fullsized":"2024-04-27T01:10:31.193722+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:14.831100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":34,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.209142+0000","last_change":"2024-04-27T01:09:51.972465+0000","last_active":"2024-04-27T01:10:31.209142+0000","last_peered":"2024-04-27T01:10:31.209142+0000","last_clean":"2024-04-27T01:10:31.209142+0000","last_became_active":"2024-04-27T01:09:51.971917+0000","last_became_peered":"2024-04-27T01:09:51.971917+0000","last_unstale":"2024-04-27T01:10:31.209142+0000","last_undegraded":"2024-04-27T01:10:31.209142+0000","last_fullsized":"2024-04-27T01:10:31.209142+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:42.551048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":62,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029172+0000","last_change":"2024-04-27T01:09:53.977524+0000","last_active":"2024-04-27T01:10:31.029172+0000","last_peered":"2024-04-27T01:10:31.029172+0000","last_clean":"2024-04-27T01:10:31.029172+0000","last_became_active":"2024-04-27T01:09:53.977141+0000","last_became_peered":"2024-04-27T01:09:53.977141+0000","last_unstale":"2024-04-27T01:10:31.029172+0000","last_undegraded":"2024-04-27T01:10:31.029172+0000","last_fullsized":"2024-04-27T01:10:31.029172+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:24.476172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"69'8","reported_seq":53,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:07.041262+0000","last_change":"2024-04-27T01:09:55.994468+0000","last_active":"2024-04-27T01:11:07.041262+0000","last_peered":"2024-04-27T01:11:07.041262+0000","last_clean":"2024-04-27T01:11:07.041262+0000","last_became_active":"2024-04-27T01:09:55.994143+0000","last_became_peered":"2024-04-27T01:09:55.994143+0000","last_unstale":"2024-04-27T01:11:07.041262+0000","last_undegraded":"2024-04-27T01:11:07.041262+0000","last_fullsized":"2024-04-27T01:11:07.041262+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:43:23.714459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028520+0000","last_change":"2024-04-27T01:09:57.996696+0000","last_active":"2024-04-27T01:10:31.028520+0000","last_peered":"2024-04-27T01:10:31.028520+0000","last_clean":"2024-04-27T01:10:31.028520+0000","last_became_active":"2024-04-27T01:09:57.996331+0000","last_became_peered":"2024-04-27T01:09:57.996331+0000","last_unstale":"2024-04-27T01:10:31.028520+0000","last_undegraded":"2024-04-27T01:10:31.028520+0000","last_fullsized":"2024-04-27T01:10:31.028520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:05:40.417633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202647+0000","last_change":"2024-04-27T01:09:51.981032+0000","last_active":"2024-04-27T01:10:31.202647+0000","last_peered":"2024-04-27T01:10:31.202647+0000","last_clean":"2024-04-27T01:10:31.202647+0000","last_became_active":"2024-04-27T01:09:51.980554+0000","last_became_peered":"2024-04-27T01:09:51.980554+0000","last_unstale":"2024-04-27T01:10:31.202647+0000","last_undegraded":"2024-04-27T01:10:31.202647+0000","last_fullsized":"2024-04-27T01:10:31.202647+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:07:13.815364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'30","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200900+0000","last_change":"2024-04-27T01:09:54.096814+0000","last_active":"2024-04-27T01:10:31.200900+0000","last_peered":"2024-04-27T01:10:31.200900+0000","last_clean":"2024-04-27T01:10:31.200900+0000","last_became_active":"2024-04-27T01:09:54.096423+0000","last_became_peered":"2024-04-27T01:09:54.096423+0000","last_unstale":"2024-04-27T01:10:31.200900+0000","last_undegraded":"2024-04-27T01:10:31.200900+0000","last_fullsized":"2024-04-27T01:10:31.200900+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:53:25.325208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028454+0000","last_change":"2024-04-27T01:09:55.992285+0000","last_active":"2024-04-27T01:10:31.028454+0000","last_peered":"2024-04-27T01:10:31.028454+0000","last_clean":"2024-04-27T01:10:31.028454+0000","last_became_active":"2024-04-27T01:09:55.992003+0000","last_became_peered":"2024-04-27T01:09:55.992003+0000","last_unstale":"2024-04-27T01:10:31.028454+0000","last_undegraded":"2024-04-27T01:10:31.028454+0000","last_fullsized":"2024-04-27T01:10:31.028454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:29:04.778310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194046+0000","last_change":"2024-04-27T01:09:58.003940+0000","last_active":"2024-04-27T01:10:31.194046+0000","last_peered":"2024-04-27T01:10:31.194046+0000","last_clean":"2024-04-27T01:10:31.194046+0000","last_became_active":"2024-04-27T01:09:58.003730+0000","last_became_peered":"2024-04-27T01:09:58.003730+0000","last_unstale":"2024-04-27T01:10:31.194046+0000","last_undegraded":"2024-04-27T01:10:31.194046+0000","last_fullsized":"2024-04-27T01:10:31.194046+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:50:57.443520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"69'8","reported_seq":50,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:07.041548+0000","last_change":"2024-04-27T01:09:55.986323+0000","last_active":"2024-04-27T01:11:07.041548+0000","last_peered":"2024-04-27T01:11:07.041548+0000","last_clean":"2024-04-27T01:11:07.041548+0000","last_became_active":"2024-04-27T01:09:55.986022+0000","last_became_peered":"2024-04-27T01:09:55.986022+0000","last_unstale":"2024-04-27T01:11:07.041548+0000","last_undegraded":"2024-04-27T01:11:07.041548+0000","last_fullsized":"2024-04-27T01:11:07.041548+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:18:53.897875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200812+0000","last_change":"2024-04-27T01:09:51.974389+0000","last_active":"2024-04-27T01:10:31.200812+0000","last_peered":"2024-04-27T01:10:31.200812+0000","last_clean":"2024-04-27T01:10:31.200812+0000","last_became_active":"2024-04-27T01:09:51.974094+0000","last_became_peered":"2024-04-27T01:09:51.974094+0000","last_unstale":"2024-04-27T01:10:31.200812+0000","last_undegraded":"2024-04-27T01:10:31.200812+0000","last_fullsized":"2024-04-27T01:10:31.200812+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:49:15.151744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":54,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.038572+0000","last_change":"2024-04-27T01:09:53.983723+0000","last_active":"2024-04-27T01:10:31.038572+0000","last_peered":"2024-04-27T01:10:31.038572+0000","last_clean":"2024-04-27T01:10:31.038572+0000","last_became_active":"2024-04-27T01:09:53.983380+0000","last_became_peered":"2024-04-27T01:09:53.983380+0000","last_unstale":"2024-04-27T01:10:31.038572+0000","last_undegraded":"2024-04-27T01:10:31.038572+0000","last_fullsized":"2024-04-27T01:10:31.038572+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:48.838115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.200964+0000","last_change":"2024-04-27T01:09:58.492072+0000","last_active":"2024-04-27T01:10:31.200964+0000","last_peered":"2024-04-27T01:10:31.200964+0000","last_clean":"2024-04-27T01:10:31.200964+0000","last_became_active":"2024-04-27T01:09:58.491675+0000","last_became_peered":"2024-04-27T01:09:58.491675+0000","last_unstale":"2024-04-27T01:10:31.200964+0000","last_undegraded":"2024-04-27T01:10:31.200964+0000","last_fullsized":"2024-04-27T01:10:31.200964+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:19:27.606788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.195371+0000","last_change":"2024-04-27T01:09:55.990732+0000","last_active":"2024-04-27T01:10:31.195371+0000","last_peered":"2024-04-27T01:10:31.195371+0000","last_clean":"2024-04-27T01:10:31.195371+0000","last_became_active":"2024-04-27T01:09:55.990078+0000","last_became_peered":"2024-04-27T01:09:55.990078+0000","last_unstale":"2024-04-27T01:10:31.195371+0000","last_undegraded":"2024-04-27T01:10:31.195371+0000","last_fullsized":"2024-04-27T01:10:31.195371+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:15:04.385756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028929+0000","last_change":"2024-04-27T01:09:51.974876+0000","last_active":"2024-04-27T01:10:31.028929+0000","last_peered":"2024-04-27T01:10:31.028929+0000","last_clean":"2024-04-27T01:10:31.028929+0000","last_became_active":"2024-04-27T01:09:51.974577+0000","last_became_peered":"2024-04-27T01:09:51.974577+0000","last_unstale":"2024-04-27T01:10:31.028929+0000","last_undegraded":"2024-04-27T01:10:31.028929+0000","last_fullsized":"2024-04-27T01:10:31.028929+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:06:31.526042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'17","reported_seq":60,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029141+0000","last_change":"2024-04-27T01:09:53.981121+0000","last_active":"2024-04-27T01:10:31.029141+0000","last_peered":"2024-04-27T01:10:31.029141+0000","last_clean":"2024-04-27T01:10:31.029141+0000","last_became_active":"2024-04-27T01:09:53.980643+0000","last_became_peered":"2024-04-27T01:09:53.980643+0000","last_unstale":"2024-04-27T01:10:31.029141+0000","last_undegraded":"2024-04-27T01:10:31.029141+0000","last_fullsized":"2024-04-27T01:10:31.029141+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:09.835870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.030903+0000","last_change":"2024-04-27T01:09:58.001560+0000","last_active":"2024-04-27T01:10:31.030903+0000","last_peered":"2024-04-27T01:10:31.030903+0000","last_clean":"2024-04-27T01:10:31.030903+0000","last_became_active":"2024-04-27T01:09:58.001241+0000","last_became_peered":"2024-04-27T01:09:58.001241+0000","last_unstale":"2024-04-27T01:10:31.030903+0000","last_undegraded":"2024-04-27T01:10:31.030903+0000","last_fullsized":"2024-04-27T01:10:31.030903+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:15:24.574837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.040017+0000","last_change":"2024-04-27T01:09:55.994476+0000","last_active":"2024-04-27T01:10:31.040017+0000","last_peered":"2024-04-27T01:10:31.040017+0000","last_clean":"2024-04-27T01:10:31.040017+0000","last_became_active":"2024-04-27T01:09:55.993751+0000","last_became_peered":"2024-04-27T01:09:55.993751+0000","last_unstale":"2024-04-27T01:10:31.040017+0000","last_undegraded":"2024-04-27T01:10:31.040017+0000","last_fullsized":"2024-04-27T01:10:31.040017+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:21:57.091034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.194581+0000","last_change":"2024-04-27T01:09:51.975073+0000","last_active":"2024-04-27T01:10:31.194581+0000","last_peered":"2024-04-27T01:10:31.194581+0000","last_clean":"2024-04-27T01:10:31.194581+0000","last_became_active":"2024-04-27T01:09:51.974368+0000","last_became_peered":"2024-04-27T01:09:51.974368+0000","last_unstale":"2024-04-27T01:10:31.194581+0000","last_undegraded":"2024-04-27T01:10:31.194581+0000","last_fullsized":"2024-04-27T01:10:31.194581+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:40:27.867034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":81,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029257+0000","last_change":"2024-04-27T01:10:22.475719+0000","last_active":"2024-04-27T01:10:31.029257+0000","last_peered":"2024-04-27T01:10:31.029257+0000","last_clean":"2024-04-27T01:10:31.029257+0000","last_became_active":"2024-04-27T01:10:22.468557+0000","last_became_peered":"2024-04-27T01:10:22.468557+0000","last_unstale":"2024-04-27T01:10:31.029257+0000","last_undegraded":"2024-04-27T01:10:31.029257+0000","last_fullsized":"2024-04-27T01:10:31.029257+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:59:47.022142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.029500+0000","last_change":"2024-04-27T01:09:57.994354+0000","last_active":"2024-04-27T01:10:31.029500+0000","last_peered":"2024-04-27T01:10:31.029500+0000","last_clean":"2024-04-27T01:10:31.029500+0000","last_became_active":"2024-04-27T01:09:57.994072+0000","last_became_peered":"2024-04-27T01:09:57.994072+0000","last_unstale":"2024-04-27T01:10:31.029500+0000","last_undegraded":"2024-04-27T01:10:31.029500+0000","last_fullsized":"2024-04-27T01:10:31.029500+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:56.972946+0000","last_clean_scrub_stamp":"2024-04-27T01:09:56.972946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:56:45.195389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"69'8","reported_seq":49,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:11:02.040265+0000","last_change":"2024-04-27T01:09:55.989649+0000","last_active":"2024-04-27T01:11:02.040265+0000","last_peered":"2024-04-27T01:11:02.040265+0000","last_clean":"2024-04-27T01:11:02.040265+0000","last_became_active":"2024-04-27T01:09:55.989303+0000","last_became_peered":"2024-04-27T01:09:55.989303+0000","last_unstale":"2024-04-27T01:11:02.040265+0000","last_undegraded":"2024-04-27T01:11:02.040265+0000","last_fullsized":"2024-04-27T01:11:02.040265+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:54.964949+0000","last_clean_scrub_stamp":"2024-04-27T01:09:54.964949+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:32:17.454412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.202271+0000","last_change":"2024-04-27T01:09:51.978202+0000","last_active":"2024-04-27T01:10:31.202271+0000","last_peered":"2024-04-27T01:10:31.202271+0000","last_clean":"2024-04-27T01:10:31.202271+0000","last_became_active":"2024-04-27T01:09:51.977897+0000","last_became_peered":"2024-04-27T01:09:51.977897+0000","last_unstale":"2024-04-27T01:10:31.202271+0000","last_undegraded":"2024-04-27T01:10:31.202271+0000","last_fullsized":"2024-04-27T01:10:31.202271+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:50.957960+0000","last_clean_scrub_stamp":"2024-04-27T01:09:50.957960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:39:39.844375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":44,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-04-27T01:10:31.028807+0000","last_change":"2024-04-27T01:09:53.980407+0000","last_active":"2024-04-27T01:10:31.028807+0000","last_peered":"2024-04-27T01:10:31.028807+0000","last_clean":"2024-04-27T01:10:31.028807+0000","last_became_active":"2024-04-27T01:09:53.980029+0000","last_became_peered":"2024-04-27T01:09:53.980029+0000","last_unstale":"2024-04-27T01:10:31.028807+0000","last_undegraded":"2024-04-27T01:10:31.028807+0000","last_fullsized":"2024-04-27T01:10:31.028807+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T01:09:52.961196+0000","last_clean_scrub_stamp":"2024-04-27T01:09:52.961196+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:02.744677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":67,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201301,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32468,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712684,"statfs":{"total":95995035648,"available":95961788416,"internally_reserved":0,"allocated":1323008,"data_stored":874698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32444,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712708,"statfs":{"total":95995035648,"available":95961812992,"internally_reserved":0,"allocated":1298432,"data_stored":873600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":284687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32476,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712676,"statfs":{"total":95995035648,"available":95961780224,"internally_reserved":0,"allocated":1327104,"data_stored":875387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":716800,"data_stored":284049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":753664,"data_stored":285565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986211,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":737280,"data_stored":284830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724574,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713272,"statfs":{"total":95995035648,"available":95962390528,"internally_reserved":0,"allocated":716800,"data_stored":284137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T01:11:17.420 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-27T01:11:17.420 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-27T01:11:17.420 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-27T01:11:17.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a shell --fsid ea378f3c-0431-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-27T01:11:17.902 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/mon.a/config 2024-04-27T01:11:18.025 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:17 smithi162 ceph-mon[37680]: from='client.34646 v1:172.21.15.162:0/3064791565' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:11:18.026 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:17 smithi162 ceph-mon[26714]: from='client.34646 v1:172.21.15.162:0/3064791565' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:11:18.026 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 podman[72139]: 2024-04-27 01:11:17.74997537 +0000 UTC m=+1.538745816 container init 7c13e184d0c7282dd5cecf024cd1f6a969dfbacda025cb339094daf6c2c77da1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:18.026 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 podman[72139]: 2024-04-27 01:11:17.752964567 +0000 UTC m=+1.541735021 container start 7c13e184d0c7282dd5cecf024cd1f6a969dfbacda025cb339094daf6c2c77da1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:18.026 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:17.764Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-27T01:11:18.026 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:17.765Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-27T01:11:18.026 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:17.766Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.162 port=9094 2024-04-27T01:11:18.026 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:17.766Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-27T01:11:18.026 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:17.791Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-27T01:11:18.026 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:17.791Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-27T01:11:18.027 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:17.794Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-27T01:11:18.027 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:17.794Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-27T01:11:18.027 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 bash[72139]: 7c13e184d0c7282dd5cecf024cd1f6a969dfbacda025cb339094daf6c2c77da1 2024-04-27T01:11:18.027 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:17 smithi162 systemd[1]: Started Ceph alertmanager.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:18.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:17 smithi181 ceph-mon[30237]: from='client.34646 v1:172.21.15.162:0/3064791565' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-27T01:11:18.299 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:18] ENGINE Bus STOPPING 2024-04-27T01:11:18.299 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-27T01:11:18.299 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:18] ENGINE Bus STOPPED 2024-04-27T01:11:18.299 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:18] ENGINE Bus STARTING 2024-04-27T01:11:18.299 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:18] ENGINE Serving on http://:::9283 2024-04-27T01:11:18.299 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:18] ENGINE Bus STARTED 2024-04-27T01:11:19.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:18 smithi181 ceph-mon[30237]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:19.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:18 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:18 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:18 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:18 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:18 smithi181 ceph-mon[30237]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-27T01:11:19.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:18 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:18 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:18 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:11:19.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:18 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[37680]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:19.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[37680]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-27T01:11:19.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:11:19.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[26714]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[26714]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-27T01:11:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:11:19.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:18 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:19.826 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.836+0000 7fb77a98d700 1 -- 172.21.15.162:0/3898912558 >> v1:172.21.15.162:6790/0 conn(0x7fb77410ae70 legacy=0x7fb77410d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:19.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.836+0000 7fb77a98d700 1 -- 172.21.15.162:0/3898912558 shutdown_connections 2024-04-27T01:11:19.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.836+0000 7fb77a98d700 1 -- 172.21.15.162:0/3898912558 >> 172.21.15.162:0/3898912558 conn(0x7fb774076220 msgr2=0x7fb774076630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:19.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.836+0000 7fb77a98d700 1 -- 172.21.15.162:0/3898912558 shutdown_connections 2024-04-27T01:11:19.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.836+0000 7fb77a98d700 1 -- 172.21.15.162:0/3898912558 wait complete. 2024-04-27T01:11:19.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.837+0000 7fb77a98d700 1 Processor -- start 2024-04-27T01:11:19.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.837+0000 7fb77a98d700 1 -- start start 2024-04-27T01:11:19.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.837+0000 7fb77a98d700 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb7741a7080 con 0x7fb77410ea70 2024-04-27T01:11:19.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.837+0000 7fb77a98d700 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb7741a82a0 con 0x7fb77410ae70 2024-04-27T01:11:19.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.837+0000 7fb77a98d700 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fb7741a94e0 con 0x7fb774108100 2024-04-27T01:11:19.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.837+0000 7fb773fff700 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fb774108100 0x7fb774107420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:48830/0 (socket says 172.21.15.162:48830) 2024-04-27T01:11:19.829 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.837+0000 7fb778f2a700 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb77410ea70 0x7fb7741a5740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:41258/0 (socket says 172.21.15.162:41258) 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.837+0000 7fb773fff700 1 -- 172.21.15.162:0/3286911223 learned_addr learned my addr 172.21.15.162:0/3286911223 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500079503 0 0) 0x7fb7741a94e0 con 0x7fb774108100 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb75c0031f0 con 0x7fb774108100 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3166387186 0 0) 0x7fb7741a7080 con 0x7fb77410ea70 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7741a94e0 con 0x7fb77410ea70 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2692825154 0 0) 0x7fb7741a82a0 con 0x7fb77410ae70 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7741a7080 con 0x7fb77410ae70 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1485619549 0 0) 0x7fb75c0031f0 con 0x7fb774108100 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7741a82a0 con 0x7fb774108100 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fb7680013d0 con 0x7fb774108100 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4138443293 0 0) 0x7fb7741a94e0 con 0x7fb77410ea70 2024-04-27T01:11:19.830 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb75c0031f0 con 0x7fb77410ea70 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fb76c001af0 con 0x7fb77410ea70 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1040300923 0 0) 0x7fb7741a7080 con 0x7fb77410ae70 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7741a94e0 con 0x7fb77410ae70 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fb764004510 con 0x7fb77410ae70 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 455678955 0 0) 0x7fb7741a82a0 con 0x7fb774108100 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 >> v1:172.21.15.181:6789/0 conn(0x7fb77410ae70 legacy=0x7fb7741a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 >> v1:172.21.15.162:6789/0 conn(0x7fb77410ea70 legacy=0x7fb7741a5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7741aa720 con 0x7fb774108100 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb77a98d700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7741a72b0 con 0x7fb774108100 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb77a98d700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7741a7770 con 0x7fb774108100 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fb768001570 con 0x7fb774108100 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.838+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fb768005bf0 con 0x7fb774108100 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.839+0000 7fb77a98d700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb760005120 con 0x7fb774108100 2024-04-27T01:11:19.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.840+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7fb768003930 con 0x7fb774108100 2024-04-27T01:11:19.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.840+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(73..73 src has 1..73) ==== 6259+0+0 (unknown 1072321943 0 0) 0x7fb768089490 con 0x7fb774108100 2024-04-27T01:11:19.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:19.844+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fb768005bf0 con 0x7fb774108100 2024-04-27T01:11:19.889 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:11:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:11:19.890 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:19.767Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000845984s 2024-04-27T01:11:20.078 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:20.088+0000 7fb77a98d700 1 -- 172.21.15.162:0/3286911223 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb760005410 con 0x7fb774108100 2024-04-27T01:11:20.079 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:20.089+0000 7fb7717fa700 1 -- 172.21.15.162:0/3286911223 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7fb768056bd0 con 0x7fb774108100 2024-04-27T01:11:20.079 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-27T01:11:20.079 INFO:teuthology.orchestra.run.smithi162.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-27T01:11:20.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:20.091+0000 7fb77a98d700 1 -- 172.21.15.162:0/3286911223 >> v1:172.21.15.162:6800/624703537 conn(0x7fb75c06ef70 legacy=0x7fb75c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:20.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:20.091+0000 7fb77a98d700 1 -- 172.21.15.162:0/3286911223 >> v1:172.21.15.162:6790/0 conn(0x7fb774108100 legacy=0x7fb774107420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:11:20.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:20.091+0000 7fb77a98d700 1 -- 172.21.15.162:0/3286911223 shutdown_connections 2024-04-27T01:11:20.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:20.091+0000 7fb77a98d700 1 -- 172.21.15.162:0/3286911223 >> 172.21.15.162:0/3286911223 conn(0x7fb774076220 msgr2=0x7fb774108da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:11:20.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:20.091+0000 7fb77a98d700 1 -- 172.21.15.162:0/3286911223 shutdown_connections 2024-04-27T01:11:20.081 INFO:teuthology.orchestra.run.smithi162.stderr:2024-04-27T01:11:20.091+0000 7fb77a98d700 1 -- 172.21.15.162:0/3286911223 wait complete. 2024-04-27T01:11:20.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:19 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:11:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:19 smithi181 ceph-mon[30237]: Deploying daemon grafana.a on smithi181 2024-04-27T01:11:20.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:19 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:11:20.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:19 smithi162 ceph-mon[37680]: Deploying daemon grafana.a on smithi181 2024-04-27T01:11:20.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:19 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-27T01:11:20.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:19 smithi162 ceph-mon[26714]: Deploying daemon grafana.a on smithi181 2024-04-27T01:11:20.848 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-27T01:11:20.849 INFO:tasks.cephadm:Setup complete, yielding 2024-04-27T01:11:20.850 INFO:teuthology.run_tasks:Running task workunit... 2024-04-27T01:11:20.863 INFO:tasks.workunit:Pulling workunits from ref b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:11:20.863 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-27T01:11:20.863 DEBUG:teuthology.orchestra.run.smithi162:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-27T01:11:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:11:20.880 INFO:teuthology.orchestra.run.smithi162.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-27T01:11:20.881 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-27T01:11:20.938 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-27T01:11:20.939 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-27T01:11:20.995 INFO:tasks.workunit:timeout=3h 2024-04-27T01:11:20.995 INFO:tasks.workunit:cleanup=True 2024-04-27T01:11:20.995 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T01:11:21.053 INFO:tasks.workunit.client.0.smithi162.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-27T01:11:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:20 smithi181 ceph-mon[30237]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3286911223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T01:11:21.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:20 smithi162 ceph-mon[37680]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:21.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3286911223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T01:11:21.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:20 smithi162 ceph-mon[26714]: pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:21.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3286911223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-27T01:11:22.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:22 smithi181 ceph-mon[30237]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:22.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:22 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:22.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:11:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:11:22.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:22 smithi162 ceph-mon[37680]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:22.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:22 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:22.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:22 smithi162 ceph-mon[26714]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:22.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:22 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:23.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:23.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:24.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:24 smithi162 ceph-mon[37680]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:24.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:24 smithi162 ceph-mon[26714]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:24.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:24 smithi181 ceph-mon[30237]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:26.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:26 smithi181 ceph-mon[30237]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:26.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:26 smithi162 ceph-mon[26714]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:26.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:26 smithi162 ceph-mon[37680]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:27.511 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 systemd[1]: Starting Ceph grafana.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:11:27.892 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 podman[60953]: 2024-04-27 01:11:27.500227248 +0000 UTC m=+0.018434794 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-04-27T01:11:27.893 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 podman[60953]: 2024-04-27 01:11:27.617744797 +0000 UTC m=+0.135952381 container create 8811dd0e2dd43ec89a3670da8d322d304ec7ba48b980259409f8d80d8b32ba25 (image=quay.io/ceph/grafana:9.4.12, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-27T01:11:27.893 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 podman[60953]: 2024-04-27 01:11:27.80335324 +0000 UTC m=+0.321560779 container init 8811dd0e2dd43ec89a3670da8d322d304ec7ba48b980259409f8d80d8b32ba25 (image=quay.io/ceph/grafana:9.4.12, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-27T01:11:27.893 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 podman[60953]: 2024-04-27 01:11:27.807906228 +0000 UTC m=+0.326113767 container start 8811dd0e2dd43ec89a3670da8d322d304ec7ba48b980259409f8d80d8b32ba25 (image=quay.io/ceph/grafana:9.4.12, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-27T01:11:28.088 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:27 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:27.771Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004430424s 2024-04-27T01:11:28.144 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 bash[60953]: 8811dd0e2dd43ec89a3670da8d322d304ec7ba48b980259409f8d80d8b32ba25 2024-04-27T01:11:28.144 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 systemd[1]: Started Ceph grafana.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954633138Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954808524Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.95481966Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954822993Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954825693Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954828304Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954830856Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954833379Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954835807Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954838913Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954841884Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-27T01:11:28.145 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954844767Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-27T01:11:28.146 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954847206Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-27T01:11:28.146 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954849797Z level=info msg="Path Home" path=/usr/share/grafana 2024-04-27T01:11:28.146 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954852368Z level=info msg="Path Data" path=/var/lib/grafana 2024-04-27T01:11:28.146 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954854727Z level=info msg="Path Logs" path=/var/log/grafana 2024-04-27T01:11:28.146 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954857179Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-04-27T01:11:28.147 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954859677Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-04-27T01:11:28.147 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=settings t=2024-04-27T01:11:27.954862218Z level=info msg="App mode production" 2024-04-27T01:11:28.147 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=sqlstore t=2024-04-27T01:11:27.954903787Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-04-27T01:11:28.147 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=sqlstore t=2024-04-27T01:11:27.954914261Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-27T01:11:28.147 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.955813518Z level=info msg="Starting DB migrations" 2024-04-27T01:11:28.147 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.955895129Z level=info msg="Executing migration" id="create migration_log table" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.956500286Z level=info msg="Executing migration" id="create user table" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.956922547Z level=info msg="Executing migration" id="add unique index user.login" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.957749085Z level=info msg="Executing migration" id="add unique index user.email" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.95860731Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.959472005Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.960461177Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.961515702Z level=info msg="Executing migration" id="create user table v2" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.962311609Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.962823711Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.963688224Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.964527678Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.965379237Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.96625847Z level=info msg="Executing migration" id="Update user table charset" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.967126942Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-04-27T01:11:28.148 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.968072641Z level=info msg="Executing migration" id="Add missing user data" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.968774004Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.969624488Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.970537567Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.971118709Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.976471549Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.977034032Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.977608941Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.980257254Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.983061757Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.985925813Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.988712098Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.991601265Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.994604558Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-04-27T01:11:28.149 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:27.997365318Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.000179846Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.002764327Z level=info msg="Executing migration" id="create temp_user v2" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.005591564Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.008580045Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.011322424Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.014007094Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.016621116Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.019550279Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.0220815Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.024637611Z level=info msg="Executing migration" id="create star table" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.027267187Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-04-27T01:11:28.150 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.029910061Z level=info msg="Executing migration" id="create org table v1" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.030381734Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.033137038Z level=info msg="Executing migration" id="create org_user table v1" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.035969504Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.036586348Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.039420318Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.04199205Z level=info msg="Executing migration" id="Update org table charset" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.044384337Z level=info msg="Executing migration" id="Update org_user table charset" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.044709799Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.047640157Z level=info msg="Executing migration" id="create dashboard table" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.050281702Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.052832749Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.053467293Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.05423029Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.055187672Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.056070326Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-04-27T01:11:28.151 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.057357541Z level=info msg="Executing migration" id="create dashboard v2" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.058181802Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.058997345Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.059806547Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.060169589Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.061125084Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.061915835Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.062768017Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.063333659Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.064155966Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.065063185Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.065821132Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.066253008Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.067095722Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-04-27T01:11:28.152 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.067966844Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.06897003Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.069811044Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.070544195Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.071503417Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.072218088Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.072534184Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.072967231Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.073403924Z level=info msg="Executing migration" id="Update dashboard title length" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.073748219Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.074279273Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.075121072Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.076232205Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.077154826Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.078119089Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-27T01:11:28.153 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.078825029Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.07918809Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.082022508Z level=info msg="Executing migration" id="Add check_sum column" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.082612349Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.08471678Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.087650062Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.088204033Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.091052322Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.093778966Z level=info msg="Executing migration" id="create data_source table" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.096412012Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.096894893Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.099645103Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.102530429Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.105191887Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.106503582Z level=info msg="Executing migration" id="create data_source table v2" 2024-04-27T01:11:28.154 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.107290607Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.108222781Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.108861064Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.109255869Z level=info msg="Executing migration" id="Add column with_credentials" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.110047355Z level=info msg="Executing migration" id="Add secure json data column" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.110848395Z level=info msg="Executing migration" id="Update data_source table charset" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.111165071Z level=info msg="Executing migration" id="Update initial version to 1" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.1114619Z level=info msg="Executing migration" id="Add read_only data column" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.112043128Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.112339352Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.112802502Z level=info msg="Executing migration" id="Add uid column" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.113415394Z level=info msg="Executing migration" id="Update uid value" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.113749015Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.114257144Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.115051305Z level=info msg="Executing migration" id="create api_key table" 2024-04-27T01:11:28.155 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.115987443Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.116606653Z level=info msg="Executing migration" id="add index api_key.key" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.117579867Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.118307447Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.11911753Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.119552233Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.119972375Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.121236611Z level=info msg="Executing migration" id="create api_key table v2" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.121659955Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.122096747Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.122519428Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.122943427Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.123249254Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-04-27T01:11:28.156 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.123651149Z level=info msg="Executing migration" id="Update api_key table charset" 2024-04-27T01:11:28.157 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.12394582Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-04-27T01:11:28.157 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.124532147Z level=info msg="Executing migration" id="Add service account foreign key" 2024-04-27T01:11:28.157 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.125139755Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-04-27T01:11:28.157 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.125432541Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-04-27T01:11:28.157 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.126071384Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-04-27T01:11:28.157 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.126667557Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.127064324Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.127408486Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.127824171Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.128248218Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.128680461Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.129095055Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.129367864Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.129675812Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.130339965Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.130982531Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.131290745Z level=info msg="Executing migration" id="create quota table v1" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.131728235Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-27T01:11:28.158 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.132156423Z level=info msg="Executing migration" id="Update quota table charset" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.132431672Z level=info msg="Executing migration" id="create plugin_setting table" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.132843589Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.133259859Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.133889765Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.134190786Z level=info msg="Executing migration" id="create session table" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.134621557Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.134927655Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.135205861Z level=info msg="Executing migration" id="create playlist table v2" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.135615915Z level=info msg="Executing migration" id="create playlist item table v2" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.136006631Z level=info msg="Executing migration" id="Update playlist table charset" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.136281234Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.136557057Z level=info msg="Executing migration" id="drop preferences table v2" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.136867805Z level=info msg="Executing migration" id="drop preferences table v3" 2024-04-27T01:11:28.159 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.137149532Z level=info msg="Executing migration" id="create preferences table v3" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.137566861Z level=info msg="Executing migration" id="Update preferences table charset" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.137867403Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.138481607Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.138822159Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.139437813Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.140107487Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.140397052Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.140917172Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.141431167Z level=info msg="Executing migration" id="create alert table v1" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.142256971Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.14307737Z level=info msg="Executing migration" id="add index alert state" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.144041499Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.144787422Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-04-27T01:11:28.160 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.145620081Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-27T01:11:28.161 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.146590219Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-27T01:11:28.161 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.147305132Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-27T01:11:28.161 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.148955834Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-04-27T01:11:28.161 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.149920445Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-27T01:11:28.161 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.150666182Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-04-27T01:11:28.161 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.151563554Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-04-27T01:11:28.161 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.152402872Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-04-27T01:11:28.161 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.155328749Z level=info msg="Executing migration" id="Add column is_default" 2024-04-27T01:11:28.161 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.158057012Z level=info msg="Executing migration" id="Add column frequency" 2024-04-27T01:11:28.395 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.160726084Z level=info msg="Executing migration" id="Add column send_reminder" 2024-04-27T01:11:28.395 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.163419249Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-04-27T01:11:28.395 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.166140696Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-04-27T01:11:28.395 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.168862488Z level=info msg="Executing migration" id="Update alert table charset" 2024-04-27T01:11:28.395 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.171720299Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-04-27T01:11:28.395 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.17436934Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-04-27T01:11:28.395 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.177131808Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-27T01:11:28.395 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.180070583Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-04-27T01:11:28.396 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.182914227Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-04-27T01:11:28.396 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.185727192Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-27T01:11:28.396 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.188343231Z level=info msg="Executing migration" id="Add for to alert table" 2024-04-27T01:11:28.396 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.191031521Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-04-27T01:11:28.396 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.193844708Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-04-27T01:11:28.396 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.196684838Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-04-27T01:11:28.396 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.199192966Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-04-27T01:11:28.396 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.20189522Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-04-27T01:11:28.396 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.204318756Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-04-27T01:11:28.396 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.207185959Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-04-27T01:11:28.397 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.209704957Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-04-27T01:11:28.397 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.212251745Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-04-27T01:11:28.397 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.215019122Z level=info msg="Executing migration" id="create annotation table v5" 2024-04-27T01:11:28.397 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.217704882Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.220314196Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.223030476Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.225687161Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.228250032Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.230913523Z level=info msg="Executing migration" id="Update annotation table charset" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.233647365Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.236316863Z level=info msg="Executing migration" id="Drop category_id index" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.238866744Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.241694484Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.244364324Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.246960729Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.247697023Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-27T01:11:28.398 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.249715261Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.250556603Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.251545623Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.25217348Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.253042013Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.253468812Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.254264231Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.255089353Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.25589476Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.256731059Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.257529637Z level=info msg="Executing migration" id="Add epoch_end column" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.258324177Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.259133456Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-04-27T01:11:28.399 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.259886843Z level=info msg="Executing migration" id="Move region to single row" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.260679065Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.26159765Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.262262798Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.263223273Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.263991489Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.264783618Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.265747834Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.266509546Z level=info msg="Executing migration" id="create test_data table" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.267319172Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.267746475Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.270683397Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.273247594Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-04-27T01:11:28.400 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.275999564Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.278677679Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.281599881Z level=info msg="Executing migration" id="create team table" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.283987091Z level=info msg="Executing migration" id="add index team.org_id" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.286744956Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.289422288Z level=info msg="Executing migration" id="create team member table" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.292046032Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.294897978Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.297612552Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.300245179Z level=info msg="Executing migration" id="Add column email to team table" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.303059611Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.305804344Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.30816953Z level=info msg="Executing migration" id="create dashboard acl table" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.310888946Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.313778922Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-27T01:11:28.401 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.316312034Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.319009624Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.31984598Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.320599079Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.321528826Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.322241342Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.323165233Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.323838274Z level=info msg="Executing migration" id="create tag table" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.324679503Z level=info msg="Executing migration" id="add index tag.key_value" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.325485584Z level=info msg="Executing migration" id="create login attempt table" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.325907984Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.32674964Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.327721463Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-27T01:11:28.402 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.330160702Z level=info msg="Executing migration" id="create login_attempt v2" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.33055894Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.331564766Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.332363988Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.33303536Z level=info msg="Executing migration" id="create user auth table" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.333399315Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.333843127Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.33425796Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.335119173Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.335999057Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.336894919Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.337772345Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.338679228Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.339573765Z level=info msg="Executing migration" id="create server_lock table" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.339990942Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-04-27T01:11:28.403 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.34089556Z level=info msg="Executing migration" id="create user auth token table" 2024-04-27T01:11:28.404 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.341742863Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-04-27T01:11:28.404 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.342726372Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-04-27T01:11:28.404 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.343223733Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-04-27T01:11:28.404 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.344097473Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-04-27T01:11:28.404 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.345071742Z level=info msg="Executing migration" id="create cache_data table" 2024-04-27T01:11:28.404 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.34608742Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-04-27T01:11:28.404 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.346642006Z level=info msg="Executing migration" id="create short_url table v1" 2024-04-27T01:11:28.404 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.347540888Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-04-27T01:11:28.404 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.35027945Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-04-27T01:11:28.404 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.353271555Z level=info msg="Executing migration" id="delete alert_definition table" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.353613035Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.356518038Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.359406982Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.362024143Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.362338962Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.365284152Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.367966092Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.370877136Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.371331037Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.374236368Z level=info msg="Executing migration" id="drop alert_definition table" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.376959168Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-04-27T01:11:28.405 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.379699368Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.380148983Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.382958916Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.383433472Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.385796303Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.388439434Z level=info msg="Executing migration" id="create alert_instance table" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.388909451Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.391581274Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.392084059Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.394239655Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.396805087Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.397251638Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.401716178Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-04-27T01:11:28.406 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.405980833Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-27T01:11:28.407 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.408556028Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-04-27T01:11:28.407 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.409050394Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-04-27T01:11:28.761 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.411735773Z level=info msg="Executing migration" id="create alert_rule table" 2024-04-27T01:11:28.761 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.414295472Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-04-27T01:11:28.761 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.417120549Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-04-27T01:11:28.761 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.417678402Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.420442339Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.4229398Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.423925127Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.42484412Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.425765654Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.426638331Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.427485116Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.428451044Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.429388389Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.430171974Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.431158769Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.432110321Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-04-27T01:11:28.762 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.432983456Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-04-27T01:11:28.763 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.433930718Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-27T01:11:28.763 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.434687501Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-27T01:11:28.763 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.435181604Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-27T01:11:28.763 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.436065729Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.437229832Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.438208799Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.439217381Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.440219377Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.441199332Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.442038067Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.442915991Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.443941104Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.444885024Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.446146073Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.44713885Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.448222746Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.449079141Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.449907501Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-04-27T01:11:28.764 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.45091255Z level=info msg="Executing migration" id="create provenance_type table" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.451335346Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.452267099Z level=info msg="Executing migration" id="create alert_image table" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.453078675Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.453902414Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.454256922Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.457230031Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.459904389Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.46006196Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.46250957Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.46292309Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.465591094Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.465729592Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.465843398Z level=info msg="alerts found to migrate" alerts=0 2024-04-27T01:11:28.765 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.466298915Z level=info msg="Executing migration" id="create library_element table v1" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.468306907Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.471025629Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.471461896Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.474251024Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.476892655Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.479636205Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.480116771Z level=info msg="Executing migration" id="create data_keys table" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.482742218Z level=info msg="Executing migration" id="create secrets table" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.485517954Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.491076267Z level=info msg="Executing migration" id="add name column into data_keys" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.492165068Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.494697125Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.500269012Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-04-27T01:11:28.766 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.505911594Z level=info msg="Executing migration" id="create kv_store table v1" 2024-04-27T01:11:28.767 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.506345798Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-04-27T01:11:28.767 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.506804244Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-04-27T01:11:28.767 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.507152951Z level=info msg="Executing migration" id="create permission table" 2024-04-27T01:11:28.767 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.507575339Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-04-27T01:11:28.767 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.508047954Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-04-27T01:11:28.767 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.508538102Z level=info msg="Executing migration" id="create role table" 2024-04-27T01:11:28.767 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.508978568Z level=info msg="Executing migration" id="add column display_name" 2024-04-27T01:11:28.767 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.510081438Z level=info msg="Executing migration" id="add column group_name" 2024-04-27T01:11:28.767 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.511133419Z level=info msg="Executing migration" id="add index role.org_id" 2024-04-27T01:11:28.768 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.511614041Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-04-27T01:11:28.768 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.512103146Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-04-27T01:11:28.768 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.512619034Z level=info msg="Executing migration" id="create team role table" 2024-04-27T01:11:28.768 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.513009908Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-04-27T01:11:28.768 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.5134266Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-04-27T01:11:28.768 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.513878701Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-04-27T01:11:28.768 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.514294853Z level=info msg="Executing migration" id="create user role table" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.514687779Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.515113017Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.515543484Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.515959381Z level=info msg="Executing migration" id="create builtin role table" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.516385842Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.519101722Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.519649569Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.520752854Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.521214774Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.523876877Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.52432527Z level=info msg="Executing migration" id="add unique index role.uid" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.524805671Z level=info msg="Executing migration" id="create seed assignment table" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.525171742Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.52558693Z level=info msg="Executing migration" id="add column hidden to role table" 2024-04-27T01:11:28.769 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.52669091Z level=info msg="Executing migration" id="create query_history table v1" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.5270699Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.527504544Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.527802912Z level=info msg="Executing migration" id="teams permissions migration" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.528178748Z level=info msg="Executing migration" id="dashboard permissions" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.528657243Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.529015315Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.529293242Z level=info msg="Executing migration" id="alerting notification permissions" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.529649532Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.530023866Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.530442876Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.531587085Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.531880554Z level=info msg="Executing migration" id="create correlation table v1" 2024-04-27T01:11:28.770 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.532275417Z level=info msg="Executing migration" id="add index correlations.uid" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.535024604Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.53568172Z level=info msg="Executing migration" id="add correlation config column" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.53681881Z level=info msg="Executing migration" id="create entity_events table" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.537214683Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.537599906Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.537682076Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.537955813Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.538034021Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.538274452Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.538706485Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.541006194Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.543574074Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-27T01:11:28.771 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.546238071Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.546709467Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.549516197Z level=info msg="Executing migration" id="Drop public config table" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.552201547Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.554821241Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.555279349Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.556076718Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.556921446Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.560836194Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.562029098Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.56316216Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.564076709Z level=info msg="Executing migration" id="add share column" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.56541637Z level=info msg="Executing migration" id="create default alerting folders" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.565841466Z level=info msg="Executing migration" id="create file table" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.566628845Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-04-27T01:11:28.772 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.567595671Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.568130858Z level=info msg="Executing migration" id="create file_meta table" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.568556138Z level=info msg="Executing migration" id="file table idx: path key" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.569269893Z level=info msg="Executing migration" id="set path collation in file table" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.570002105Z level=info msg="Executing migration" id="managed permissions migration" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.570397117Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.570731733Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.571171244Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.572351982Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.573266612Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.573996148Z level=info msg="Executing migration" id="update group index for alert rules" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.574397505Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.575191539Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.576155358Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-04-27T01:11:28.773 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.577528334Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.578688933Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.57964236Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.592607739Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.593385514Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.593856109Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.597968193Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.598843562Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.59953376Z level=info msg="Executing migration" id="create folder table" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.599980864Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.600811736Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.601608508Z level=info msg="Executing migration" id="Update folder title length" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.604668059Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-04-27T01:11:28.774 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=migrator t=2024-04-27T01:11:28.605286506Z level=info msg="migrations completed" performed=485 skipped=0 duration=649.411851ms 2024-04-27T01:11:28.775 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=sqlstore t=2024-04-27T01:11:28.605688948Z level=info msg="Created default organization" 2024-04-27T01:11:28.775 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=plugin.finder t=2024-04-27T01:11:28.640850639Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-04-27T01:11:28.775 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=secrets t=2024-04-27T01:11:28.641144799Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-04-27T01:11:28.775 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=query_data t=2024-04-27T01:11:28.644277073Z level=info msg="Query Service initialization" 2024-04-27T01:11:28.775 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=live.push_http t=2024-04-27T01:11:28.646723485Z level=info msg="Live Push Gateway initialization" 2024-04-27T01:11:29.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:28 smithi162 ceph-mon[26714]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:29.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:28 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:28 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:28 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:28 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:28 smithi162 ceph-mon[37680]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:29.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:28 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:28 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:28 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:28 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.090 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:28 smithi162 systemd[1]: Starting Ceph node-exporter.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:11:29.249 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-mon[30237]: pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:29.249 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.249 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.249 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.249 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:28 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:29.511 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=infra.usagestats.collector t=2024-04-27T01:11:29.259500211Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-04-27T01:11:29.511 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=provisioning.datasources t=2024-04-27T01:11:29.261455922Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-27T01:11:29.511 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=provisioning.datasources t=2024-04-27T01:11:29.272664357Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-04-27T01:11:29.511 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=provisioning.alerting t=2024-04-27T01:11:29.283597716Z level=info msg="starting to provision alerting" 2024-04-27T01:11:29.511 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=provisioning.alerting t=2024-04-27T01:11:29.283613078Z level=info msg="finished to provision alerting" 2024-04-27T01:11:29.511 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=grafanaStorageLogger t=2024-04-27T01:11:29.283727969Z level=info msg="storage starting" 2024-04-27T01:11:29.511 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=ngalert.state.manager t=2024-04-27T01:11:29.284033211Z level=info msg="Warming state cache for startup" 2024-04-27T01:11:29.512 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=http.server t=2024-04-27T01:11:29.28567727Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-04-27T01:11:29.512 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=ngalert.state.manager t=2024-04-27T01:11:29.287253874Z level=info msg="State cache has been initialized" states=0 duration=3.219555ms 2024-04-27T01:11:29.512 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=ticker t=2024-04-27T01:11:29.287305871Z level=info msg=starting first_tick=2024-04-27T01:11:30Z 2024-04-27T01:11:29.512 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=ngalert.multiorg.alertmanager t=2024-04-27T01:11:29.287319907Z level=info msg="starting MultiOrg Alertmanager" 2024-04-27T01:11:29.512 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=sqlstore.transactions t=2024-04-27T01:11:29.301893024Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-27T01:11:29.512 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=sqlstore.transactions t=2024-04-27T01:11:29.312116523Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-04-27T01:11:29.512 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=sqlstore.transactions t=2024-04-27T01:11:29.323415197Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-04-27T01:11:29.512 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=sqlstore.transactions t=2024-04-27T01:11:29.333598153Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-04-27T01:11:29.512 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=sqlstore.transactions t=2024-04-27T01:11:29.343903182Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-04-27T01:11:29.588 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:29 smithi162 bash[72680]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-04-27T01:11:29.959 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:11:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:11:29.959 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:29 smithi162 bash[72680]: Getting image source signatures 2024-04-27T01:11:29.959 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:29 smithi162 bash[72680]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-04-27T01:11:29.959 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:29 smithi162 bash[72680]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-04-27T01:11:29.959 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:29 smithi162 bash[72680]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-04-27T01:11:30.214 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:29 smithi181 ceph-mon[30237]: Deploying daemon node-exporter.a on smithi162 2024-04-27T01:11:30.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:29 smithi162 ceph-mon[37680]: Deploying daemon node-exporter.a on smithi162 2024-04-27T01:11:30.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:29 smithi162 ceph-mon[26714]: Deploying daemon node-exporter.a on smithi162 2024-04-27T01:11:31.241 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:30 smithi162 ceph-mon[37680]: pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:31.241 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 bash[72680]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-04-27T01:11:31.241 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:30 smithi162 ceph-mon[26714]: pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:31.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:30 smithi181 ceph-mon[30237]: pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:31.586 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 bash[72680]: Writing manifest to image destination 2024-04-27T01:11:31.839 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 podman[72680]: 2024-04-27 01:11:31.507873253 +0000 UTC m=+2.435665673 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-04-27T01:11:31.839 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 podman[72680]: 2024-04-27 01:11:31.606322467 +0000 UTC m=+2.534114887 container create f34e1c8822b17aa863c70511f51f8c4a77a04c23a8a8374574fdc8558d57131c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:31.839 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 podman[72680]: 2024-04-27 01:11:31.792496025 +0000 UTC m=+2.720288446 container init f34e1c8822b17aa863c70511f51f8c4a77a04c23a8a8374574fdc8558d57131c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:31.839 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 podman[72680]: 2024-04-27 01:11:31.795401173 +0000 UTC m=+2.723193592 container start f34e1c8822b17aa863c70511f51f8c4a77a04c23a8a8374574fdc8558d57131c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:31.839 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.799Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-27T01:11:31.839 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.799Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-27T01:11:31.840 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.799Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-27T01:11:31.840 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.799Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-27T01:11:31.840 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-27T01:11:31.840 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-27T01:11:31.840 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-27T01:11:31.840 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=arp 2024-04-27T01:11:31.840 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-27T01:11:31.840 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-27T01:11:31.840 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=edac 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-27T01:11:31.841 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=os 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-27T01:11:31.842 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=stat 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=time 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=uname 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-27T01:11:31.843 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-27T01:11:31.844 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-27T01:11:31.844 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-27T01:11:31.844 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a[72729]: ts=2024-04-27T01:11:31.800Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-27T01:11:32.274 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 bash[72680]: f34e1c8822b17aa863c70511f51f8c4a77a04c23a8a8374574fdc8558d57131c 2024-04-27T01:11:32.275 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:11:31 smithi162 systemd[1]: Started Ceph node-exporter.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:32.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:11:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:11:32.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:32 smithi181 ceph-mon[30237]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:32.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:32 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:32 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:32.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:32 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:32 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:32 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[37680]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:32.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:32.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[26714]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:32 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:33.280 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:33 smithi181 bash[61235]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-04-27T01:11:33.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:33 smithi162 ceph-mon[37680]: Deploying daemon node-exporter.b on smithi181 2024-04-27T01:11:33.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:33.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:33 smithi162 ceph-mon[26714]: Deploying daemon node-exporter.b on smithi181 2024-04-27T01:11:33.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:33.744 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:33 smithi181 ceph-mon[30237]: Deploying daemon node-exporter.b on smithi181 2024-04-27T01:11:33.745 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:34.010 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:33 smithi181 bash[61235]: Getting image source signatures 2024-04-27T01:11:34.011 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:33 smithi181 bash[61235]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-04-27T01:11:34.011 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:33 smithi181 bash[61235]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-04-27T01:11:34.011 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:33 smithi181 bash[61235]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-04-27T01:11:34.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:34 smithi162 ceph-mon[37680]: pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:34.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:34 smithi162 ceph-mon[26714]: pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:34 smithi181 ceph-mon[30237]: pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:35.761 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:35 smithi181 bash[61235]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-04-27T01:11:35.761 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:35 smithi181 bash[61235]: Writing manifest to image destination 2024-04-27T01:11:36.207 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:35 smithi181 podman[61235]: 2024-04-27 01:11:35.867472484 +0000 UTC m=+2.743620423 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-04-27T01:11:36.207 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:35 smithi181 podman[61235]: 2024-04-27 01:11:35.974408859 +0000 UTC m=+2.850556799 container create 435a34b6823d6e76b0dedafcd34657b2a6ebe7406265a385463c702b3bd82d85 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-27T01:11:36.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-mon[30237]: pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:36.511 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 podman[61235]: 2024-04-27 01:11:36.218747533 +0000 UTC m=+3.094895473 container init 435a34b6823d6e76b0dedafcd34657b2a6ebe7406265a385463c702b3bd82d85 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-27T01:11:36.511 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 podman[61235]: 2024-04-27 01:11:36.22147832 +0000 UTC m=+3.097626268 container start 435a34b6823d6e76b0dedafcd34657b2a6ebe7406265a385463c702b3bd82d85 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-27T01:11:36.511 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.225Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-27T01:11:36.511 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.225Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-27T01:11:36.511 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.225Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-27T01:11:36.511 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.225Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-27T01:11:36.511 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.225Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-27T01:11:36.511 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-27T01:11:36.512 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-27T01:11:36.512 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=arp 2024-04-27T01:11:36.512 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-27T01:11:36.512 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-27T01:11:36.512 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-27T01:11:36.512 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-27T01:11:36.512 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-27T01:11:36.512 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=edac 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-27T01:11:36.513 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=os 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-27T01:11:36.514 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=stat 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=time 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=uname 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-27T01:11:36.515 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-27T01:11:36.516 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-27T01:11:36.516 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-27T01:11:36.516 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b[61284]: ts=2024-04-27T01:11:36.226Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-27T01:11:36.516 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 bash[61235]: 435a34b6823d6e76b0dedafcd34657b2a6ebe7406265a385463c702b3bd82d85 2024-04-27T01:11:36.516 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:11:36 smithi181 systemd[1]: Started Ceph node-exporter.b for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:36.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:36 smithi162 ceph-mon[26714]: pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:36.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:36 smithi162 ceph-mon[37680]: pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:37.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:37.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:37 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:37.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:37 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:37 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:37 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:37 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:37 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:37.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:37 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:38 smithi181 ceph-mon[30237]: pgmap v36: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:38.580 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:38 smithi162 ceph-mon[37680]: pgmap v36: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:38.580 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:38 smithi162 ceph-mon[26714]: pgmap v36: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:40.089 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:11:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:11:40.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:40 smithi181 ceph-mon[30237]: pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:40.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:40 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:40.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:40 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:40.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:40 smithi162 ceph-mon[26714]: pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:40.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:40 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:40.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:40 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:40.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:40 smithi162 ceph-mon[37680]: pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:40.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:40 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:40.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:40 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.208 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:11:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:11:42.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:41 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:41 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:41 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:41 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:42.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:41 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:11:42.208 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:41 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:42.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:11:42.243 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.243 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:42.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:42.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:11:42.244 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:41 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:42 smithi181 ceph-mon[30237]: pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:42 smithi181 ceph-mon[30237]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-27T01:11:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:42 smithi181 ceph-mon[30237]: Reconfiguring daemon alertmanager.a on smithi162 2024-04-27T01:11:43.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:42 smithi162 ceph-mon[37680]: pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:42 smithi162 ceph-mon[37680]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-27T01:11:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:42 smithi162 ceph-mon[37680]: Reconfiguring daemon alertmanager.a on smithi162 2024-04-27T01:11:43.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:42 smithi162 ceph-mon[26714]: pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:43.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:42 smithi162 ceph-mon[26714]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-27T01:11:43.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:42 smithi162 ceph-mon[26714]: Reconfiguring daemon alertmanager.a on smithi162 2024-04-27T01:11:44.198 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:44.198 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:44.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:45.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:44 smithi181 ceph-mon[30237]: pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:45.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:44 smithi162 ceph-mon[37680]: pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:45.331 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:44 smithi162 ceph-mon[26714]: pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:45.331 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:45 smithi162 systemd[1]: Stopping Ceph alertmanager.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:11:45.588 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:45 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[72222]: ts=2024-04-27T01:11:45.341Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-27T01:11:45.589 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:45 smithi162 podman[73453]: 2024-04-27 01:11:45.37197181 +0000 UTC m=+0.147993243 container died 7c13e184d0c7282dd5cecf024cd1f6a969dfbacda025cb339094daf6c2c77da1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:45.973 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:45 smithi162 podman[73453]: 2024-04-27 01:11:45.724511924 +0000 UTC m=+0.500533360 container cleanup 7c13e184d0c7282dd5cecf024cd1f6a969dfbacda025cb339094daf6c2c77da1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:45.973 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:45 smithi162 bash[73453]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a 2024-04-27T01:11:46.261 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:45 smithi162 podman[73465]: 2024-04-27 01:11:45.983891845 +0000 UTC m=+0.616976494 container remove 7c13e184d0c7282dd5cecf024cd1f6a969dfbacda025cb339094daf6c2c77da1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:46.261 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 podman[73465]: 2024-04-27 01:11:46.093172123 +0000 UTC m=+0.726256772 volume remove 6ad03e9c425108c5ad494939eb382ed9f5929ae4d0c19d7464033af8e547cede 2024-04-27T01:11:46.588 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-27T01:11:46.588 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 systemd[1]: Stopped Ceph alertmanager.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:46.589 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 systemd[1]: Starting Ceph alertmanager.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:11:46.589 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 podman[73566]: 2024-04-27 01:11:46.401038721 +0000 UTC m=+0.018504289 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-04-27T01:11:46.589 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 podman[73566]: 2024-04-27 01:11:46.510880696 +0000 UTC m=+0.128346248 volume create d32d3a76b94a4ea7dc935ba9bb1bc2c775e14569cd4bb47ac0649b4c8b1b713b 2024-04-27T01:11:46.903 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 podman[73566]: 2024-04-27 01:11:46.628019234 +0000 UTC m=+0.245484784 container create a798feaa6f73ad454dd68307082716b40072e035e27e34c22e66e1ec9089fff7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:47.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:46 smithi181 ceph-mon[30237]: pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:47.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:46 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:47.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-mon[37680]: pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:47.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:47.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-mon[26714]: pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:47.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:11:47.339 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 podman[73566]: 2024-04-27 01:11:46.914230456 +0000 UTC m=+0.531696006 container init a798feaa6f73ad454dd68307082716b40072e035e27e34c22e66e1ec9089fff7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:47.339 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 podman[73566]: 2024-04-27 01:11:46.916793388 +0000 UTC m=+0.534258939 container start a798feaa6f73ad454dd68307082716b40072e035e27e34c22e66e1ec9089fff7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:47.340 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:11:46.928Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-27T01:11:47.340 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:11:46.928Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-27T01:11:47.340 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:11:46.929Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.162 port=9094 2024-04-27T01:11:47.340 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:11:46.930Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-27T01:11:47.340 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:11:46.952Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-27T01:11:47.340 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:11:46.952Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-27T01:11:47.340 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:11:46.956Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-27T01:11:47.340 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:46 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:11:46.956Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-27T01:11:47.340 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:47 smithi162 bash[73566]: a798feaa6f73ad454dd68307082716b40072e035e27e34c22e66e1ec9089fff7 2024-04-27T01:11:47.340 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:47 smithi162 systemd[1]: Started Ceph alertmanager.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:48.168 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:48 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:48.168 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:48 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:48.169 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:48 smithi181 ceph-mon[30237]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-27T01:11:48.169 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:48 smithi181 ceph-mon[30237]: pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:48.169 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:48 smithi181 ceph-mon[30237]: Reconfiguring daemon prometheus.a on smithi181 2024-04-27T01:11:48.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:48.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:48.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-mon[37680]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-27T01:11:48.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-mon[37680]: pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:48.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-mon[37680]: Reconfiguring daemon prometheus.a on smithi181 2024-04-27T01:11:48.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:48.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:48.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-mon[26714]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-27T01:11:48.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-mon[26714]: pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:48.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-mon[26714]: Reconfiguring daemon prometheus.a on smithi181 2024-04-27T01:11:49.338 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:48 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:11:48.931Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000754328s 2024-04-27T01:11:50.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:11:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:11:50.229 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-mon[30237]: pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:50.510 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 systemd[1]: Stopping Ceph prometheus.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:11:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:50 smithi162 ceph-mon[37680]: pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:50 smithi162 ceph-mon[26714]: pgmap v42: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:50.826 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-27T01:11:50.826 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-27T01:11:50.826 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-27T01:11:50.826 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-27T01:11:50.826 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-27T01:11:50.826 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-27T01:11:50.826 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-27T01:11:50.826 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-27T01:11:50.826 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-27T01:11:50.827 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-27T01:11:50.827 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.577Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-27T01:11:50.827 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[58971]: ts=2024-04-27T01:11:50.578Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-27T01:11:50.827 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 podman[62187]: 2024-04-27 01:11:50.587487218 +0000 UTC m=+0.144034692 container died fb877d9ea8947c99b71c940707eb0a14a7ee945d9ad972ea23bc66da38f2caa2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:51.200 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 podman[62187]: 2024-04-27 01:11:50.935628503 +0000 UTC m=+0.492175975 container cleanup fb877d9ea8947c99b71c940707eb0a14a7ee945d9ad972ea23bc66da38f2caa2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:51.200 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:50 smithi181 bash[62187]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a 2024-04-27T01:11:51.510 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 podman[62201]: 2024-04-27 01:11:51.211686708 +0000 UTC m=+0.625976985 container remove fb877d9ea8947c99b71c940707eb0a14a7ee945d9ad972ea23bc66da38f2caa2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:51.511 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-27T01:11:51.511 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 systemd[1]: Stopped Ceph prometheus.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:51.511 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@prometheus.a.service: Consumed 1.098s CPU time. 2024-04-27T01:11:51.511 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 systemd[1]: Starting Ceph prometheus.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:11:51.813 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 podman[62295]: 2024-04-27 01:11:51.472904754 +0000 UTC m=+0.019138265 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-04-27T01:11:51.813 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 podman[62295]: 2024-04-27 01:11:51.596080463 +0000 UTC m=+0.142313962 container create 8b221f51564304e296a84629833ab84319976c265bc591502cef6741287c06eb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:51.813 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 podman[62295]: 2024-04-27 01:11:51.797905952 +0000 UTC m=+0.344139451 container init 8b221f51564304e296a84629833ab84319976c265bc591502cef6741287c06eb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:51.813 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 podman[62295]: 2024-04-27 01:11:51.800411061 +0000 UTC m=+0.346644561 container start 8b221f51564304e296a84629833ab84319976c265bc591502cef6741287c06eb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-27T01:11:52.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:51] ENGINE Bus STOPPING 2024-04-27T01:11:52.089 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-27T01:11:52.089 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STOPPED 2024-04-27T01:11:52.089 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STARTING 2024-04-27T01:11:52.110 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.824Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-27T01:11:52.110 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.824Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-27T01:11:52.110 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.824Z caller=main.go:570 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi181 (none))" 2024-04-27T01:11:52.110 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.824Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.824Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.825Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.825Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.827Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.827Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.828Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.828Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.449µs 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.828Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.829Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.829Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.829Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.95µs wal_replay_duration=1.421405ms wbl_replay_duration=130ns total_replay_duration=1.457849ms 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.830Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.830Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.830Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-27T01:11:52.111 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.846Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.372511ms db_storage=676ns remote_storage=1.507µs web_handler=344ns query_engine=522ns scrape=417.224µs scrape_sd=80.539µs notify=26.897µs notify_sd=22.82µs rules=14.598962ms tracing=5.726µs 2024-04-27T01:11:52.112 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.846Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-27T01:11:52.112 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:11:51.846Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-27T01:11:52.112 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 bash[62295]: 8b221f51564304e296a84629833ab84319976c265bc591502cef6741287c06eb 2024-04-27T01:11:52.112 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:11:51 smithi181 systemd[1]: Started Ceph prometheus.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:11:52.421 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Serving on http://:::9283 2024-04-27T01:11:52.421 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STARTED 2024-04-27T01:11:52.421 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STOPPING 2024-04-27T01:11:52.421 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-27T01:11:52.421 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STOPPED 2024-04-27T01:11:52.421 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STARTING 2024-04-27T01:11:52.421 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Serving on http://:::9283 2024-04-27T01:11:52.421 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STARTED 2024-04-27T01:11:52.422 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STOPPING 2024-04-27T01:11:52.422 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-27T01:11:52.422 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STOPPED 2024-04-27T01:11:52.422 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STARTING 2024-04-27T01:11:52.422 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Serving on http://:::9283 2024-04-27T01:11:52.422 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:11:52] ENGINE Bus STARTED 2024-04-27T01:11:52.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162:9093"}]: dispatch 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:11:53.190 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162:9093"}]: dispatch 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:11:53.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T01:11:53.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:52 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:53.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162:9093"}]: dispatch 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:53.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:52 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:11:53.984 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:53 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:11:53.984 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:53 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162:9093"}]: dispatch 2024-04-27T01:11:53.984 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:53 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:11:53.984 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:53 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T01:11:53.985 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:53 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:11:53.985 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:53 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T01:11:53.985 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:54.261 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:11:54.261 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162:9093"}]: dispatch 2024-04-27T01:11:54.261 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:11:54.261 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T01:11:54.261 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:11:54.261 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T01:11:54.261 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:54.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-27T01:11:54.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi162:9093"}]: dispatch 2024-04-27T01:11:54.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-27T01:11:54.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi181.front.sepia.ceph.com:9095"}]: dispatch 2024-04-27T01:11:54.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-27T01:11:54.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi181.front.sepia.ceph.com:3000"}]: dispatch 2024-04-27T01:11:54.262 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:11:55.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:54 smithi162 ceph-mon[37680]: pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:55.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:54 smithi162 ceph-mon[26714]: pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:54 smithi181 ceph-mon[30237]: pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:56 smithi162 ceph-mon[37680]: pgmap v45: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:56 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:56.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:56 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:56 smithi162 ceph-mon[26714]: pgmap v45: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:56 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:56 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:56 smithi181 ceph-mon[30237]: pgmap v45: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:11:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:56 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:56.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:56 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:57.338 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:11:56 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:11:56.934Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003853929s 2024-04-27T01:11:58.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:58 smithi181 ceph-mon[30237]: pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:58.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:58 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:58.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:58 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:58.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:58 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:58.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:58 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:11:58.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:11:58 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:58.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[37680]: pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:11:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:58.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[26714]: pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:11:58.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:58.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:11:58.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:11:58.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:11:58.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:11:58 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:12:00.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:11:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:11:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:12:00.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:00 smithi181 ceph-mon[30237]: pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:00 smithi162 ceph-mon[37680]: pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:00.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:00 smithi162 ceph-mon[26714]: pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:01.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:01 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:01.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:01 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:02.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:01 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:02.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:12:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:12:02.783 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:02 smithi181 ceph-mon[30237]: pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:02.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:02 smithi162 ceph-mon[37680]: pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:02.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:02 smithi162 ceph-mon[26714]: pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:03.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:03.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:04.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:04 smithi162 ceph-mon[37680]: pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:04 smithi162 ceph-mon[26714]: pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:04 smithi181 ceph-mon[30237]: pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:06 smithi162 ceph-mon[37680]: pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:06.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:06 smithi162 ceph-mon[26714]: pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:07.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:06 smithi181 ceph-mon[30237]: pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:08.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:08 smithi162 ceph-mon[37680]: pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:08.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:08 smithi162 ceph-mon[26714]: pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:09.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:08 smithi181 ceph-mon[30237]: pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:10.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:12:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:12:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:12:10.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:10 smithi162 ceph-mon[37680]: pgmap v52: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:10.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:10 smithi162 ceph-mon[26714]: pgmap v52: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:11.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:10 smithi181 ceph-mon[30237]: pgmap v52: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:12.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:12:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:12:12.526 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:12 smithi162 ceph-mon[26714]: pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:12.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:12 smithi162 ceph-mon[37680]: pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:12 smithi181 ceph-mon[30237]: pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:13.533 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:13.788 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:13.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:14.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:14 smithi162 ceph-mon[37680]: pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:14 smithi162 ceph-mon[26714]: pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:15.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:14 smithi181 ceph-mon[30237]: pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:16.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:16 smithi162 ceph-mon[37680]: pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:16.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:16.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:16 smithi162 ceph-mon[26714]: pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:16.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:16 smithi181 ceph-mon[30237]: pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:18 smithi162 ceph-mon[37680]: pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:18.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:18 smithi162 ceph-mon[26714]: pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:18 smithi181 ceph-mon[30237]: pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:20.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:12:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:12:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:12:20.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:20 smithi162 ceph-mon[26714]: pgmap v57: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:20.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:20 smithi162 ceph-mon[37680]: pgmap v57: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:21.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:20 smithi181 ceph-mon[30237]: pgmap v57: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:22.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:12:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:12:22.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:22 smithi162 ceph-mon[37680]: pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:22.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:22 smithi162 ceph-mon[26714]: pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:22 smithi181 ceph-mon[30237]: pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:23.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:24.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:25.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:24 smithi162 ceph-mon[37680]: pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:25.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:24 smithi162 ceph-mon[26714]: pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:25.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:24 smithi181 ceph-mon[30237]: pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:27.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:26 smithi162 ceph-mon[37680]: pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:27.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:26 smithi162 ceph-mon[26714]: pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:27.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:26 smithi181 ceph-mon[30237]: pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:29.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:28 smithi162 ceph-mon[37680]: pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:29.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:28 smithi162 ceph-mon[26714]: pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:29.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:28 smithi181 ceph-mon[30237]: pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:30.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:12:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:12:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:12:31.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:30 smithi162 ceph-mon[26714]: pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:31.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:30 smithi162 ceph-mon[37680]: pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:31.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:30 smithi181 ceph-mon[30237]: pgmap v62: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:31.713 INFO:tasks.workunit.client.0.smithi162.stderr:Note: switching to 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'. 2024-04-27T01:12:31.713 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-04-27T01:12:31.713 INFO:tasks.workunit.client.0.smithi162.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-27T01:12:31.713 INFO:tasks.workunit.client.0.smithi162.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-27T01:12:31.713 INFO:tasks.workunit.client.0.smithi162.stderr:state without impacting any branches by switching back to a branch. 2024-04-27T01:12:31.713 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-04-27T01:12:31.713 INFO:tasks.workunit.client.0.smithi162.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-27T01:12:31.713 INFO:tasks.workunit.client.0.smithi162.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-27T01:12:31.714 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-04-27T01:12:31.714 INFO:tasks.workunit.client.0.smithi162.stderr: git switch -c 2024-04-27T01:12:31.714 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-04-27T01:12:31.714 INFO:tasks.workunit.client.0.smithi162.stderr:Or undo this operation with: 2024-04-27T01:12:31.714 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-04-27T01:12:31.714 INFO:tasks.workunit.client.0.smithi162.stderr: git switch - 2024-04-27T01:12:31.714 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-04-27T01:12:31.714 INFO:tasks.workunit.client.0.smithi162.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-27T01:12:31.714 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-04-27T01:12:31.714 INFO:tasks.workunit.client.0.smithi162.stderr:HEAD is now at b22e2ebdeb2 Merge pull request #57078 from adk3798/squid-cephadm-no-downgrade 2024-04-27T01:12:31.717 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-27T01:12:31.765 INFO:tasks.workunit.client.0.smithi162.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-27T01:12:31.767 INFO:tasks.workunit.client.0.smithi162.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-27T01:12:31.767 INFO:tasks.workunit.client.0.smithi162.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-27T01:12:32.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:32.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:32.146 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:32.404 INFO:tasks.workunit.client.0.smithi162.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-27T01:12:32.457 INFO:tasks.workunit.client.0.smithi162.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-27T01:12:32.495 INFO:tasks.workunit.client.0.smithi162.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-27T01:12:32.497 INFO:tasks.workunit.client.0.smithi162.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-27T01:12:32.497 INFO:tasks.workunit.client.0.smithi162.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-27T01:12:32.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:12:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:12:32.535 INFO:tasks.workunit.client.0.smithi162.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-27T01:12:32.538 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-04-27T01:12:32.538 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-27T01:12:32.553 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-04-27T01:12:32.554 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-04-27T01:12:32.555 DEBUG:teuthology.orchestra.run.smithi162:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b22e2ebdeb24376882b7bda2a7329c8cccc2276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-04-27T01:12:32.624 INFO:tasks.workunit.client.0.smithi162.stderr:+ parallel=1 2024-04-27T01:12:32.625 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' '' = --serial ']' 2024-04-27T01:12:32.625 INFO:tasks.workunit.client.0.smithi162.stderr:+ crimson=0 2024-04-27T01:12:32.625 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' '' = --crimson ']' 2024-04-27T01:12:32.625 INFO:tasks.workunit.client.0.smithi162.stderr:+ color= 2024-04-27T01:12:32.625 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -t 1 ']' 2024-04-27T01:12:32.625 INFO:tasks.workunit.client.0.smithi162.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-04-27T01:12:32.625 INFO:tasks.workunit.client.0.smithi162.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-27T01:12:32.625 INFO:tasks.workunit.client.0.smithi162.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-27T01:12:32.626 INFO:tasks.workunit.client.0.smithi162.stderr:+ declare -A pids 2024-04-27T01:12:32.626 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.626 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.626 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_aio 2024-04-27T01:12:32.626 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_aio' 2024-04-27T01:12:32.627 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_aio 2024-04-27T01:12:32.627 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.627 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_aio 2024-04-27T01:12:32.628 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74243 2024-04-27T01:12:32.628 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_aio on pid 74243' 2024-04-27T01:12:32.628 INFO:tasks.workunit.client.0.smithi162.stdout:test api_aio on pid 74243 2024-04-27T01:12:32.631 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74243 2024-04-27T01:12:32.631 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.631 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.631 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-04-27T01:12:32.631 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_aio_pp 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_aio_pp' 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_aio_pp 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.632 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.633 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.634 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.634 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.634 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.634 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.634 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.634 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.634 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.634 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_aio_pp 2024-04-27T01:12:32.634 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74250 2024-04-27T01:12:32.634 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_aio_pp on pid 74250' 2024-04-27T01:12:32.635 INFO:tasks.workunit.client.0.smithi162.stdout:test api_aio_pp on pid 74250 2024-04-27T01:12:32.638 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74250 2024-04-27T01:12:32.638 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_io 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_io' 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_io 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.639 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_io 2024-04-27T01:12:32.642 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74259 2024-04-27T01:12:32.643 INFO:tasks.workunit.client.0.smithi162.stdout:test api_io on pid 74259 2024-04-27T01:12:32.646 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_io on pid 74259' 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74259 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_io_pp 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_io_pp' 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_io_pp 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.647 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.648 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.649 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.650 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.651 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.652 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.652 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_io_pp 2024-04-27T01:12:32.652 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74271 2024-04-27T01:12:32.652 INFO:tasks.workunit.client.0.smithi162.stdout:test api_io_pp on pid 74271 2024-04-27T01:12:32.657 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_io_pp on pid 74271' 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74271 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_asio 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_asio' 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_asio 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.658 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.659 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.660 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.661 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.662 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.663 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.664 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.664 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.664 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.664 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.664 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_asio 2024-04-27T01:12:32.664 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74287 2024-04-27T01:12:32.664 INFO:tasks.workunit.client.0.smithi162.stdout:test api_asio on pid 74287 2024-04-27T01:12:32.672 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_asio on pid 74287' 2024-04-27T01:12:32.673 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74287 2024-04-27T01:12:32.673 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.673 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.673 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-04-27T01:12:32.673 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_list 2024-04-27T01:12:32.673 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_list' 2024-04-27T01:12:32.673 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.673 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.674 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_list 2024-04-27T01:12:32.674 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.674 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.674 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.674 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.674 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.674 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.674 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.675 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.675 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.675 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.675 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.675 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.675 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.675 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.675 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.675 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.676 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.676 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.676 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.676 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.676 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.676 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.676 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.676 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.677 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.677 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.677 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.677 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.677 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.677 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.677 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.677 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.678 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.678 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.678 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.678 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.678 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.678 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.678 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.678 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.679 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.679 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.679 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.679 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.679 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.679 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.679 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.679 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.680 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.680 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.680 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.680 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.680 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.680 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.680 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.681 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.681 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.681 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.681 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.681 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.681 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.682 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.682 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.682 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.682 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.682 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.682 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74243/exe 2024-04-27T01:12:32.683 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.683 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.683 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.683 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.683 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.683 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.683 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.684 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.685 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.686 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.686 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.686 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.686 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.686 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.686 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.686 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.687 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.687 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.687 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.687 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.687 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_list 2024-04-27T01:12:32.687 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74304 2024-04-27T01:12:32.687 INFO:tasks.workunit.client.0.smithi162.stdout:test api_list on pid 74304 2024-04-27T01:12:32.693 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_list on pid 74304' 2024-04-27T01:12:32.693 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74304 2024-04-27T01:12:32.693 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.693 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.693 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-04-27T01:12:32.693 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.693 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.693 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.693 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.693 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_lock 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_lock' 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_aio: /' 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.694 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_lock 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_aio.log 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.695 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.699 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74250/exe 2024-04-27T01:12:32.700 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.701 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.702 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_lock 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.703 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.704 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.704 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.704 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74327 2024-04-27T01:12:32.704 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.704 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_lock on pid 74327' 2024-04-27T01:12:32.704 INFO:tasks.workunit.client.0.smithi162.stdout:test api_lock on pid 74327 2024-04-27T01:12:32.708 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74327 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.709 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_aio_pp: /' 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.710 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.711 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.712 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_lock_pp 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_lock_pp' 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.713 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_lock_pp 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74259/exe 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.715 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.717 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.718 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.718 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.718 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.718 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.718 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.718 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.718 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.718 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.718 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.719 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.720 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.720 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.720 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.720 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.720 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.720 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.720 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.720 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_lock_pp 2024-04-27T01:12:32.722 INFO:tasks.workunit.client.0.smithi162.stdout:test api_lock_pp on pid 74345 2024-04-27T01:12:32.727 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74345 2024-04-27T01:12:32.727 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_lock_pp on pid 74345' 2024-04-27T01:12:32.727 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74345 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_misc 2024-04-27T01:12:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.729 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74271/exe 2024-04-27T01:12:32.729 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.729 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.729 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_misc' 2024-04-27T01:12:32.729 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_io.log 2024-04-27T01:12:32.729 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.729 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.729 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_io: /' 2024-04-27T01:12:32.729 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.729 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_misc 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.730 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.731 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.732 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.732 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.732 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.732 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.732 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.732 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.732 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.733 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.733 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.733 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.733 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.733 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.733 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.733 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.733 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.733 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.734 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.734 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.734 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.734 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.734 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.734 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.734 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-04-27T01:12:32.734 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-04-27T01:12:32.734 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-04-27T01:12:32.735 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_misc 2024-04-27T01:12:32.735 INFO:tasks.workunit.client.0.smithi162.stdout:test api_misc on pid 74364 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74364 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_misc on pid 74364' 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74364 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_misc_pp 2024-04-27T01:12:32.741 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_misc_pp' 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_misc_pp 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74287/exe 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.744 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.747 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.748 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_asio.log 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_asio: /' 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.749 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.750 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.751 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.752 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.753 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_io_pp: /' 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.754 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.755 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.755 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.755 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.755 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.755 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.755 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.755 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.755 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.755 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.756 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.756 INFO:tasks.workunit.client.0.smithi162.stdout:test api_misc_pp on pid 74395 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.759 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_misc_pp 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74395 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_misc_pp on pid 74395' 2024-04-27T01:12:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74395 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_tier_pp 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_tier_pp' 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.761 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74304/exe 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.762 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.763 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.764 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.765 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_tier_pp 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_list: /' 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74327/exe 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.766 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_list.log 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.767 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.768 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.768 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.768 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.768 INFO:tasks.workunit.client.0.smithi162.stdout:test api_tier_pp on pid 74413 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_tier_pp 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74413 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_tier_pp on pid 74413' 2024-04-27T01:12:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74413 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_pool 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_pool' 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.772 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-04-27T01:12:32.774 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.775 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_lock.log 2024-04-27T01:12:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.776 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.777 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.777 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.777 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.777 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.777 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_pool 2024-04-27T01:12:32.777 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_lock: /' 2024-04-27T01:12:32.777 INFO:tasks.workunit.client.0.smithi162.stdout:test api_pool on pid 74425 2024-04-27T01:12:32.780 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_pool 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74425 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_pool on pid 74425' 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74425 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_snapshots 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_snapshots' 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74345/exe 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_snapshots 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-04-27T01:12:32.781 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.783 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.784 INFO:tasks.workunit.client.0.smithi162.stdout:test api_snapshots on pid 74446 2024-04-27T01:12:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_snapshots 2024-04-27T01:12:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74446 2024-04-27T01:12:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_snapshots on pid 74446' 2024-04-27T01:12:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74446 2024-04-27T01:12:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.786 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_snapshots_pp 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.787 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.788 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_snapshots_pp' 2024-04-27T01:12:32.789 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_snapshots_pp 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_lock_pp: /' 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.791 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-04-27T01:12:32.792 INFO:tasks.workunit.client.0.smithi162.stdout:test api_snapshots_pp on pid 74481 2024-04-27T01:12:32.795 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_snapshots_pp 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74481 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_snapshots_pp on pid 74481' 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74481 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.796 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.797 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.797 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_stat 2024-04-27T01:12:32.797 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.797 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.797 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_stat' 2024-04-27T01:12:32.797 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-04-27T01:12:32.797 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.797 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.797 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.798 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.800 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.801 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.802 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.803 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74364/exe 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.804 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_stat 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.805 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.806 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.806 INFO:tasks.workunit.client.0.smithi162.stdout:test api_stat on pid 74534 2024-04-27T01:12:32.811 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_stat 2024-04-27T01:12:32.811 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74534 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_stat on pid 74534' 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74534 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.812 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.813 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_stat_pp 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.814 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_stat_pp' 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.815 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.816 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_stat_pp 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_misc: /' 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74413/exe 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_misc.log 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.817 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.818 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.818 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.818 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.818 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.818 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.818 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.818 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.818 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.818 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.819 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.820 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.820 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.820 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.820 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.820 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_stat_pp 2024-04-27T01:12:32.820 INFO:tasks.workunit.client.0.smithi162.stdout:test api_stat_pp on pid 74556 2024-04-27T01:12:32.825 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74556 2024-04-27T01:12:32.825 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_stat_pp on pid 74556' 2024-04-27T01:12:32.825 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74556 2024-04-27T01:12:32.825 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.825 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.825 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.825 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.826 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.826 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.826 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_tier_pp: /' 2024-04-27T01:12:32.826 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.826 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.826 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.826 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.826 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.826 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.828 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.828 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_watch_notify 2024-04-27T01:12:32.828 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_watch_notify' 2024-04-27T01:12:32.828 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-04-27T01:12:32.828 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-04-27T01:12:32.828 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.829 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.829 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.829 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-04-27T01:12:32.829 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.829 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.829 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.830 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.830 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.830 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.830 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.830 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.830 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.831 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.831 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.831 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.831 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.831 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.831 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.832 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.832 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.832 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.832 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.832 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.832 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.833 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_watch_notify 2024-04-27T01:12:32.833 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.833 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.833 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.833 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.833 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.834 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.834 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.834 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.834 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.835 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.835 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.835 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.835 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.836 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.836 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.836 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.836 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.836 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.836 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.837 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.837 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.837 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.837 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.837 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.837 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.837 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.837 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.837 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.837 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.838 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.838 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.838 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.838 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.838 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.838 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.838 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.838 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.838 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.839 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.839 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.839 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.839 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.839 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.839 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.839 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.839 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.840 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.840 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.840 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.840 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_watch_notify 2024-04-27T01:12:32.840 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74576 2024-04-27T01:12:32.840 INFO:tasks.workunit.client.0.smithi162.stdout:test api_watch_notify on pid 74576 2024-04-27T01:12:32.844 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_watch_notify on pid 74576' 2024-04-27T01:12:32.844 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74576 2024-04-27T01:12:32.844 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.844 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.844 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_watch_notify_pp 2024-04-27T01:12:32.844 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.845 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.845 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.845 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.845 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.845 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_watch_notify_pp' 2024-04-27T01:12:32.845 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.845 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.845 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.845 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.846 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.846 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.846 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.846 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.846 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.846 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.846 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.846 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.847 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.847 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.847 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.847 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.847 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.847 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.847 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.847 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.847 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.848 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.848 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.848 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.848 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.848 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.848 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.848 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.848 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.848 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.850 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.850 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.850 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_watch_notify_pp 2024-04-27T01:12:32.850 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.850 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-04-27T01:12:32.850 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_watch_notify_pp 2024-04-27T01:12:32.851 INFO:tasks.workunit.client.0.smithi162.stdout:test api_watch_notify_pp on pid 74593 2024-04-27T01:12:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74593 2024-04-27T01:12:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_watch_notify_pp on pid 74593' 2024-04-27T01:12:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74593 2024-04-27T01:12:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_cmd 2024-04-27T01:12:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.860 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_cmd' 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.861 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.862 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.862 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.862 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.862 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.862 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.862 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.862 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74395/exe 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.863 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.865 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.866 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.867 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74425/exe 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.871 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.872 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.873 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_pool.log 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_pool: /' 2024-04-27T01:12:32.874 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.875 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_cmd 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.878 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_cmd 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.879 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.880 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74640 2024-04-27T01:12:32.880 INFO:tasks.workunit.client.0.smithi162.stdout:test api_cmd on pid 74640 2024-04-27T01:12:32.886 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_cmd on pid 74640' 2024-04-27T01:12:32.886 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74640 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_misc_pp: /' 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.888 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.889 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.890 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74481/exe 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.894 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_cmd_pp 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_cmd_pp' 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.900 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.901 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.902 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.903 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_cmd_pp 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.904 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74556/exe 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.905 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.906 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.906 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.906 INFO:tasks.workunit.client.0.smithi162.stdout:test api_cmd_pp on pid 74712 2024-04-27T01:12:32.913 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.913 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.913 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_cmd_pp 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74712 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_cmd_pp on pid 74712' 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74712 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74576/exe 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.914 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.915 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74446/exe 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.916 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.917 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.918 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.919 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_service 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_service' 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.920 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-04-27T01:12:32.921 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_watch_notify: /' 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74593/exe 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.922 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.923 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.924 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.925 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-04-27T01:12:32.925 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-04-27T01:12:32.925 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-04-27T01:12:32.925 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_snapshots: /' 2024-04-27T01:12:32.925 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-04-27T01:12:32.925 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.926 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.927 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.927 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.927 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.927 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.927 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.927 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.927 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.927 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_service 2024-04-27T01:12:32.927 INFO:tasks.workunit.client.0.smithi162.stdout:test api_service on pid 74756 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_service 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74756 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_service on pid 74756' 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74756 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_stat_pp: /' 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74534/exe 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.936 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.937 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.937 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.937 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.937 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.937 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.937 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.937 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.937 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.937 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.937 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_service_pp 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_service_pp' 2024-04-27T01:12:32.938 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-04-27T01:12:32.939 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.939 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.939 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.939 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.939 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.939 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.939 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_stat.log 2024-04-27T01:12:32.939 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_stat: /' 2024-04-27T01:12:32.939 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.939 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.940 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.940 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.940 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.940 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.940 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-04-27T01:12:32.940 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.940 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.940 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.940 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.940 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_service_pp 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.942 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.944 INFO:tasks.workunit.client.0.smithi162.stdout:test api_service_pp on pid 74803 2024-04-27T01:12:32.946 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_service_pp 2024-04-27T01:12:32.946 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74803 2024-04-27T01:12:32.946 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_service_pp on pid 74803' 2024-04-27T01:12:32.946 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74803 2024-04-27T01:12:32.946 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.946 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.946 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74640/exe 2024-04-27T01:12:32.946 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.946 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.946 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.947 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.948 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_cmd.log 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_c_write_operations 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_c_write_operations' 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.949 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.950 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_c_write_operations 2024-04-27T01:12:32.950 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_cmd: /' 2024-04-27T01:12:32.950 INFO:tasks.workunit.client.0.smithi162.stdout:test api_c_write_operations on pid 74844 2024-04-27T01:12:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_c_write_operations 2024-04-27T01:12:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74844 2024-04-27T01:12:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_c_write_operations on pid 74844' 2024-04-27T01:12:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74844 2024-04-27T01:12:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-04-27T01:12:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.957 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_c_read_operations 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_c_read_operations' 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.958 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.959 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.961 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_c_read_operations 2024-04-27T01:12:32.964 INFO:tasks.workunit.client.0.smithi162.stdout:test api_c_read_operations on pid 74894 2024-04-27T01:12:32.968 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_c_read_operations 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74894 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_c_read_operations on pid 74894' 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74894 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.969 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.970 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.971 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s api_cls_remote_reads 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.974 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' api_cls_remote_reads' 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo api_cls_remote_reads 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.976 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.977 INFO:tasks.workunit.client.0.smithi162.stdout:test api_cls_remote_reads on pid 74925 2024-04-27T01:12:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=api_cls_remote_reads 2024-04-27T01:12:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74925 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test api_cls_remote_reads on pid 74925' 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74925 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s list_parallel 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' list_parallel' 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74712/exe 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:32.985 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:32.986 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.987 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.988 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:32.989 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.990 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo list_parallel 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:32.991 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.993 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74756/exe 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:32.996 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=list_parallel 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74981 2024-04-27T01:12:32.997 INFO:tasks.workunit.client.0.smithi162.stdout:test list_parallel on pid 74981 2024-04-27T01:12:33.000 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test list_parallel on pid 74981' 2024-04-27T01:12:33.000 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=74981 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s open_pools_parallel 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' open_pools_parallel' 2024-04-27T01:12:33.001 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo open_pools_parallel 2024-04-27T01:12:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.003 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.004 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.005 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.006 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74803/exe 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.007 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74844/exe 2024-04-27T01:12:33.008 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.009 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.012 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.013 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.013 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.013 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.013 INFO:tasks.workunit.client.0.smithi162.stdout:test open_pools_parallel on pid 75018 2024-04-27T01:12:33.017 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.017 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.017 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_service.log 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=open_pools_parallel 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_service: /' 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75018 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test open_pools_parallel on pid 75018' 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75018 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.018 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74925/exe 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.021 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.022 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s delete_pools_parallel 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' delete_pools_parallel' 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.023 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_service_pp: /' 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.024 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.025 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.026 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.027 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.027 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.027 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.027 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.027 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.027 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.027 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.027 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.027 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.027 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo delete_pools_parallel 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.028 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.029 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74894/exe 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.030 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.031 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=delete_pools_parallel 2024-04-27T01:12:33.032 INFO:tasks.workunit.client.0.smithi162.stdout:test delete_pools_parallel on pid 75051 2024-04-27T01:12:33.036 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75051 2024-04-27T01:12:33.036 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test delete_pools_parallel on pid 75051' 2024-04-27T01:12:33.036 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75051 2024-04-27T01:12:33.036 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s cls 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' cls' 2024-04-27T01:12:33.037 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.040 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.041 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.042 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/74981/exe 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.043 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.044 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.045 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.046 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.046 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.046 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.046 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.046 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.046 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.046 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-04-27T01:12:33.047 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.047 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.047 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.047 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.047 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.047 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.047 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.048 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.048 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.048 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.048 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.048 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.048 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.049 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.049 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.049 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.049 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.049 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.049 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.050 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.050 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.050 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.050 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.050 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.050 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.051 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.051 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.051 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.051 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75018/exe 2024-04-27T01:12:33.051 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.051 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.051 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.052 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.052 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.052 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.052 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.052 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.052 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.052 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.052 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-04-27T01:12:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo cls 2024-04-27T01:12:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-04-27T01:12:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-04-27T01:12:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_list_parallel.log 2024-04-27T01:12:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ list_parallel: /' 2024-04-27T01:12:33.054 INFO:tasks.workunit.client.0.smithi162.stdout:test cls on pid 75101 2024-04-27T01:12:33.057 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=cls 2024-04-27T01:12:33.058 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75101 2024-04-27T01:12:33.058 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test cls on pid 75101' 2024-04-27T01:12:33.058 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75101 2024-04-27T01:12:33.058 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.058 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.058 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-04-27T01:12:33.058 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.058 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.058 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.058 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.059 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s cmd 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' cmd' 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo cmd 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.060 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.061 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.062 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.063 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75051/exe 2024-04-27T01:12:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.065 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.066 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-04-27T01:12:33.067 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-04-27T01:12:33.067 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.067 INFO:tasks.workunit.client.0.smithi162.stdout:test cmd on pid 75197 2024-04-27T01:12:33.068 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=cmd 2024-04-27T01:12:33.068 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75197 2024-04-27T01:12:33.068 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test cmd on pid 75197' 2024-04-27T01:12:33.068 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75197 2024-04-27T01:12:33.068 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.068 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.068 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-04-27T01:12:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s handler_error 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' handler_error' 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo handler_error 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.070 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.071 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.071 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.071 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.071 INFO:tasks.workunit.client.0.smithi162.stdout:test handler_error on pid 75221 2024-04-27T01:12:33.072 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=handler_error 2024-04-27T01:12:33.072 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.072 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.072 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.072 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.072 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.072 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75221 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test handler_error on pid 75221' 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75221 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s io 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' io' 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo io 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.073 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.074 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.074 INFO:tasks.workunit.client.0.smithi162.stdout:test io on pid 75246 2024-04-27T01:12:33.080 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=io 2024-04-27T01:12:33.080 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75246 2024-04-27T01:12:33.080 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test io on pid 75246' 2024-04-27T01:12:33.080 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75246 2024-04-27T01:12:33.080 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.080 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.081 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.082 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s ec_io 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' ec_io' 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.083 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.084 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75101/exe 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.085 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.086 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo ec_io 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.087 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.088 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.089 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_cls.log 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.090 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.091 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.092 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_cls 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ cls: /' 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.093 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.094 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.094 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.094 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.094 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.094 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.094 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.094 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.094 INFO:tasks.workunit.client.0.smithi162.stdout:test ec_io on pid 75316 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=ec_io 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75316 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test ec_io on pid 75316' 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75316 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s list 2024-04-27T01:12:33.096 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' list' 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.097 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo list 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.098 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.100 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.100 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.100 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.100 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.100 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.100 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.100 INFO:tasks.workunit.client.0.smithi162.stdout:test list on pid 75329 2024-04-27T01:12:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=list 2024-04-27T01:12:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75329 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test list on pid 75329' 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75329 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.104 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s ec_list 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' ec_list' 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.105 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.106 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.107 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.108 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo ec_list 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.109 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.110 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.111 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.112 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.114 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75221/exe 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.115 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=ec_list 2024-04-27T01:12:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.116 INFO:tasks.workunit.client.0.smithi162.stdout:test ec_list on pid 75397 2024-04-27T01:12:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.119 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75397 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test ec_list on pid 75397' 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75397 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.120 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s misc 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_handler_error.log 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' misc' 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ handler_error: /' 2024-04-27T01:12:33.121 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_handler_error 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.122 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.123 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo misc 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.124 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.127 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.128 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=misc 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75445 2024-04-27T01:12:33.129 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test misc on pid 75445' 2024-04-27T01:12:33.130 INFO:tasks.workunit.client.0.smithi162.stdout:test misc on pid 75445 2024-04-27T01:12:33.136 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75445 2024-04-27T01:12:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-04-27T01:12:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s pool 2024-04-27T01:12:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' pool' 2024-04-27T01:12:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75246/exe 2024-04-27T01:12:33.137 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75197/exe 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo pool 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.138 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.139 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75316/exe 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.140 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.141 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.141 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.141 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.141 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.141 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.141 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.141 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.141 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.141 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.142 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.143 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_cmd 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_ec_io 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_ec_io.log 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ cmd: /' 2024-04-27T01:12:33.144 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_cmd.log 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.145 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_io 2024-04-27T01:12:33.146 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.146 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.146 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.146 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.146 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.146 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.146 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.146 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=pool 2024-04-27T01:12:33.147 INFO:tasks.workunit.client.0.smithi162.stdout:test pool on pid 75480 2024-04-27T01:12:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75480 2024-04-27T01:12:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test pool on pid 75480' 2024-04-27T01:12:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75480 2024-04-27T01:12:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_io.log 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ io: /' 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.155 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.156 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.158 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.159 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.160 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75329/exe 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s read_operations 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.161 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' read_operations' 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.162 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.163 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.163 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.163 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.163 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.163 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.163 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo read_operations 2024-04-27T01:12:33.163 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_list 2024-04-27T01:12:33.163 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.164 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.164 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.164 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.164 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.164 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.164 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.164 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.164 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.165 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.165 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.165 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.165 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.165 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.165 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.166 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.166 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.166 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.166 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.166 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.166 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.167 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.167 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.167 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.167 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.167 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.167 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.168 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.169 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.169 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.169 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.169 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.169 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.170 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.170 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.170 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.170 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.170 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.170 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.171 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.171 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.171 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.171 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.171 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.172 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.172 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.172 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.172 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.172 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.172 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.173 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.173 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.173 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.173 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.173 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.173 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.175 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.175 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.175 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.175 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.175 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.176 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.176 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.176 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ ec_io: /' 2024-04-27T01:12:33.176 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.176 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.176 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.177 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.177 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.177 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_list.log 2024-04-27T01:12:33.177 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.177 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ list: /' 2024-04-27T01:12:33.177 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.178 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.178 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.178 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.178 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.178 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.179 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.179 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.179 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.179 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.179 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.180 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.180 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.180 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.180 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.180 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.180 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.180 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.181 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.181 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.181 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.181 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.181 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.182 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75445/exe 2024-04-27T01:12:33.182 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.182 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.182 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.182 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.182 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.183 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=read_operations 2024-04-27T01:12:33.183 INFO:tasks.workunit.client.0.smithi162.stdout:test read_operations on pid 75529 2024-04-27T01:12:33.187 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75529 2024-04-27T01:12:33.187 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test read_operations on pid 75529' 2024-04-27T01:12:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75529 2024-04-27T01:12:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-04-27T01:12:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s snapshots 2024-04-27T01:12:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' snapshots' 2024-04-27T01:12:33.188 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_misc.log 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.194 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.194 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.194 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.194 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.194 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ misc: /' 2024-04-27T01:12:33.194 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo snapshots 2024-04-27T01:12:33.194 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_misc 2024-04-27T01:12:33.194 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75397/exe 2024-04-27T01:12:33.194 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=snapshots 2024-04-27T01:12:33.194 INFO:tasks.workunit.client.0.smithi162.stdout:test snapshots on pid 75553 2024-04-27T01:12:33.197 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75553 2024-04-27T01:12:33.197 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test snapshots on pid 75553' 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75553 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s watch_notify 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' watch_notify' 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.199 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.200 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ ec_list: /' 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo watch_notify 2024-04-27T01:12:33.201 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_ec_list.log 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_ec_list 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.202 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.203 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.204 INFO:tasks.workunit.client.0.smithi162.stdout:test watch_notify on pid 75573 2024-04-27T01:12:33.207 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=watch_notify 2024-04-27T01:12:33.207 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75573 2024-04-27T01:12:33.207 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test watch_notify on pid 75573' 2024-04-27T01:12:33.207 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75573 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:++ printf %25s write_operations 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:+ r=' write_operations' 2024-04-27T01:12:33.208 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ echo write_operations 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.212 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:++ awk '{print $1}' 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.213 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.215 INFO:tasks.workunit.client.0.smithi162.stdout:test write_operations on pid 75596 2024-04-27T01:12:33.225 INFO:tasks.workunit.client.0.smithi162.stderr:+ ff=write_operations 2024-04-27T01:12:33.225 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75596 2024-04-27T01:12:33.225 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo 'test write_operations on pid 75596' 2024-04-27T01:12:33.225 INFO:tasks.workunit.client.0.smithi162.stderr:+ pids[$f]=75596 2024-04-27T01:12:33.226 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.226 INFO:tasks.workunit.client.0.smithi162.stderr:+ ret=0 2024-04-27T01:12:33.226 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 1 -eq 1 ']' 2024-04-27T01:12:33.226 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:12:33.226 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74413 2024-04-27T01:12:33.226 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74413 2024-04-27T01:12:33.226 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.226 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.226 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.227 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.227 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.227 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.227 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.227 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.227 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.228 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.228 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.228 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.228 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.228 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.228 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.229 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.229 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.229 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.229 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.229 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.229 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.230 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.230 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.230 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.230 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.230 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.230 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.232 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.232 INFO:tasks.workunit.client.0.smithi162.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-04-27T01:12:33.232 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.232 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.232 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.233 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.233 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.233 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.233 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.233 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.233 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.233 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.234 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.234 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.234 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.234 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.234 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.235 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.235 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.235 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.235 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.235 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.235 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.236 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.236 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -f /etc/bashrc ']' 2024-04-27T01:12:33.236 INFO:tasks.workunit.client.0.smithi162.stderr:+ . /etc/bashrc 2024-04-27T01:12:33.236 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.236 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.236 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.237 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.237 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.237 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75480/exe 2024-04-27T01:12:33.237 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.237 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.238 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -z '' ']' 2024-04-27T01:12:33.238 INFO:tasks.workunit.client.0.smithi162.stderr:++ BASHRCSOURCED=Y 2024-04-27T01:12:33.238 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.238 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.238 INFO:tasks.workunit.client.0.smithi162.stderr:++ shopt -q login_shell 2024-04-27T01:12:33.239 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.239 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.239 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.239 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.239 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.239 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.240 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.240 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.240 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.240 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.240 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.240 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.241 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.241 INFO:tasks.workunit.client.0.smithi162.stderr:+++ umask 2024-04-27T01:12:33.241 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.241 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.241 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' 0022 -eq 0 ']' 2024-04-27T01:12:33.241 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.242 INFO:tasks.workunit.client.0.smithi162.stderr:++ SHELL=/bin/bash 2024-04-27T01:12:33.242 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.242 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.242 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.242 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.243 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.243 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.243 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.243 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-04-27T01:12:33.243 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.243 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/bash_completion.sh 2024-04-27T01:12:33.244 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.244 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.244 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-04-27T01:12:33.244 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.244 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-04-27T01:12:33.244 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.245 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.245 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/cobbler.sh 2024-04-27T01:12:33.245 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.245 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.245 INFO:tasks.workunit.client.0.smithi162.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-04-27T01:12:33.245 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-04-27T01:12:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.247 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorgrep.sh 2024-04-27T01:12:33.247 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.247 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75529/exe 2024-04-27T01:12:33.247 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.247 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.247 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.248 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.248 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.248 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.248 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.248 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.248 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.248 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.248 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.249 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.249 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.249 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.249 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.249 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.249 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.249 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_pool.log 2024-04-27T01:12:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.250 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ pool: /' 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_pool 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ read_operations: /' 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'grep=grep --color=auto' 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'egrep=egrep --color=auto' 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorls.sh 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' '!' -t 0 ']' 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:+++ return 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-04-27T01:12:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export S_COLORS=auto 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:+++ S_COLORS=auto 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.255 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_read_operations 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.256 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75553/exe 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.257 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.258 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.259 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_snapshots 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-04-27T01:12:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+++ /usr/libexec/grepconf.sh -c 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75573/exe 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ watch_notify: /' 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/debuginfod.sh 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-04-27T01:12:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/gawk.sh 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/lang.sh 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_watch_notify 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ snapshots: /' 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_snapshots.log 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/locale 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_read_operations.log 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG_backup=en_US.UTF-8 2024-04-27T01:12:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /etc/locale.conf ']' 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -x /usr/bin/sed ']' 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:++++ export LANG=en_US.UTF-8 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:++++ LANG=en_US.UTF-8 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ LANG=en_US.UTF-8 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ unset LANG_backup config 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n '' ']' 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-04-27T01:12:33.264 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' dumb = linux ']' 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/less.sh 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:++ for i in /etc/profile.d/*.sh 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' '' ']' 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:++ . /etc/profile.d/which2.sh 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:+++++ readlink /proc/75596/exe 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:++++ basename /usr/bin/bash 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_declare='declare -f' 2024-04-27T01:12:33.265 INFO:tasks.workunit.client.0.smithi162.stderr:+++ which_opt=-f 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export which_declare 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:+++ export -f which 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset i 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:++ unset -f pathmunge 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:+ export PATH 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:+ unset rc 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph_test_neorados_write_operations 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:+ tee ceph_test_neorados_write_operations.log 2024-04-27T01:12:33.266 INFO:tasks.workunit.client.0.smithi162.stderr:+ sed 's/^/ write_operations: /' 2024-04-27T01:12:33.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:33 smithi162 ceph-mon[37680]: pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:33.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:33 smithi162 ceph-mon[26714]: pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:33.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:33 smithi181 ceph-mon[30237]: pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:12:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:38 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/167460765' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-27T01:12:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:38 smithi181 ceph-mon[30237]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:38 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/167460765' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-27T01:12:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:38 smithi162 ceph-mon[37680]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:38.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:38 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/167460765' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-27T01:12:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:38 smithi162 ceph-mon[26714]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:40.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:12:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:12:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:12:41.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2537257861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-74325-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi162-75471-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1231355398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-74360-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi162-75564-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: pgmap v66: 196 pgs: 64 unknown, 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:12:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2257886736' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-27T01:12:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: osdmap e74: 8 total, 8 up, 8 in 2024-04-27T01:12:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi162-75471-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.24658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-74325-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.24661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-74360-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi162-75564-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:40 smithi181 ceph-mon[30237]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:41.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2537257861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-74325-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi162-75471-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1231355398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-74360-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi162-75564-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: pgmap v66: 196 pgs: 64 unknown, 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2257886736' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: osdmap e74: 8 total, 8 up, 8 in 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi162-75471-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.24658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-74325-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.24661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-74360-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi162-75564-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[37680]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2537257861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-74325-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi162-75471-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1231355398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-74360-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi162-75564-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: pgmap v66: 196 pgs: 64 unknown, 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2257886736' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: osdmap e74: 8 total, 8 up, 8 in 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi162-75471-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.24658 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-74325-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.24661 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-74360-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi162-75564-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:41.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:40 smithi162 ceph-mon[26714]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:42.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: pgmap v67: 196 pgs: 44 creating+peering, 132 active+clean, 20 unknown; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-74405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1598020568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-74448-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1107003653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3493349979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-74719-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1345794867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-74726-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1404538284' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-74772-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/148729264' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-74307-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1134377238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-74351-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1194278002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-74682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3004308083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-74551-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3433423887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-74716-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1937274331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3062175585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-75054-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/741476620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/668888269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75031-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3931517720' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-74415-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: pgmap v69: 868 pgs: 44 creating+peering, 132 active+clean, 692 unknown; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1229224077' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: osdmap e75: 8 total, 8 up, 8 in 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-74448-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-74719-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-74726-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-74772-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-74307-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-74405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-74551-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34676 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-74351-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-74682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-75054-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-74716-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75031-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34769 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-74415-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi162-75471-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-74325-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-74360-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi162-75564-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2117475782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74946-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi162-75471-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi162-75564-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: osdmap e76: 8 total, 8 up, 8 in 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi162-75471-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.24877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74946-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.014 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:41 smithi181 ceph-mon[30237]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi162-75564-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [----------] Global test environment set-up. 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [----------] 12 tests from AsioRados 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (3 ms) 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2 ms) 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (3 ms) 2024-04-27T01:12:42.043 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (6 ms) 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (3 ms) 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (4 ms) 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (3 ms) 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-04-27T01:12:42.044 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (3 ms) 2024-04-27T01:12:42.045 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [----------] 12 tests from AsioRados (30 ms total) 2024-04-27T01:12:42.045 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: 2024-04-27T01:12:42.045 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [----------] Global test environment tear-down 2024-04-27T01:12:42.045 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (9011 ms total) 2024-04-27T01:12:42.045 INFO:tasks.workunit.client.0.smithi162.stdout: api_asio: [ PASSED ] 12 tests. 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: pgmap v67: 196 pgs: 44 creating+peering, 132 active+clean, 20 unknown; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-74405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1598020568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-74448-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1107003653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3493349979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-74719-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1345794867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-74726-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1404538284' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-74772-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/148729264' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-74307-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1134377238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-74351-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1194278002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-74682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3004308083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-74551-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3433423887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-74716-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1937274331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3062175585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-75054-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/741476620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: pgmap v67: 196 pgs: 44 creating+peering, 132 active+clean, 20 unknown; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:12:42.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-74405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1598020568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-74448-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1107003653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3493349979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-74719-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1345794867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-74726-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1404538284' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-74772-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/148729264' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-74307-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1134377238' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-74351-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1194278002' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-74682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3004308083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-74551-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3433423887' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-74716-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1937274331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3062175585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-75054-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/741476620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/668888269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75031-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3931517720' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-74415-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: pgmap v69: 868 pgs: 44 creating+peering, 132 active+clean, 692 unknown; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1229224077' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: osdmap e75: 8 total, 8 up, 8 in 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-74448-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-74719-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-74726-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-74772-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-74307-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-74405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-74551-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34676 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-74351-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-74682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-75054-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/668888269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75031-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3931517720' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-74415-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: pgmap v69: 868 pgs: 44 creating+peering, 132 active+clean, 692 unknown; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:12:42.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:42.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1229224077' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: osdmap e75: 8 total, 8 up, 8 in 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-74448-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-74719-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-74726-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-74772-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24667 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-74307-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34694 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-74405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-74551-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34676 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-74351-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34724 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-74682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.094 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24772 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-75054-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-74716-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75031-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34769 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-74415-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi162-75471-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-74325-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-74360-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi162-75564-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2117475782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74946-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi162-75471-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi162-75564-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: osdmap e76: 8 total, 8 up, 8 in 2024-04-27T01:12:42.095 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi162-75471-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.24877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74946-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[37680]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi162-75564-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-74716-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75031-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34769 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-74415-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi162-75471-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24658 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi162-74325-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24661 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi162-74360-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:42.096 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi162-75564-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:42.097 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2117475782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74946-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.097 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi162-75471-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:42.097 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi162-75564-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:42.097 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: osdmap e76: 8 total, 8 up, 8 in 2024-04-27T01:12:42.097 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi162-75471-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:42.097 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.24877 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74946-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:42.097 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:41 smithi162 ceph-mon[26714]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi162-75564-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:42.129 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [----------] Global test environment set-up. 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [----------] 4 tests from LibRadosService 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5040 ms) 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ OK ] LibRadosService.RegisterLate (15 ms) 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: cluster: 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: id: ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: health: HEALTH_OK 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: services: 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-04-27T01:12:42.130 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: data: 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pools: 6 pools, 132 pgs 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: objects: 230 objects, 583 KiB 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: usage: 247 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pgs: 132 active+clean 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: io: 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.131 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: cluster: 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: id: ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: health: HEALTH_OK 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: services: 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: data: 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pools: 6 pools, 132 pgs 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: objects: 230 objects, 583 KiB 2024-04-27T01:12:42.132 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: usage: 247 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pgs: 132 active+clean 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: io: 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: cluster: 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: id: ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: health: HEALTH_WARN 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 31 pool(s) do not have an application enabled 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: services: 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-04-27T01:12:42.133 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: data: 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pools: 39 pools, 1188 pgs 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: objects: 230 objects, 583 KiB 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: usage: 247 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: pgs: 85.185% pgs unknown 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 3.704% pgs not active 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 1012 unknown 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 132 active+clean 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 44 creating+peering 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:12:42.134 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: io: 2024-04-27T01:12:42.135 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: client: 655 B/s rd, 0 op/s rd, 0 op/s wr 2024-04-27T01:12:42.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-04-27T01:12:42.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] Global test environment set-up. 2024-04-27T01:12:42.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 7 tests from LibRadosList 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.ListObjects (417 ms) 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (35 ms) 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo1,foo2,foo3 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo1 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo2 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo3 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo1,foo4,foo5 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo4 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo5 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo1 2024-04-27T01:12:42.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo6,foo7 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo7 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo6 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo4 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo5 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns2:foo7 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns2:foo6 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo1 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo1 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo2 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo3 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (91 ms) 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-04-27T01:12:42.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 1 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 10 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 13 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 7 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 14 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 0 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 15 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 11 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 5 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 8 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 6 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 3 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 4 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 12 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 9 0 2024-04-27T01:12:42.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 2 0 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (68 ms) 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: x cursor=MIN 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=1 cursor=13:02547ec2:::1:head 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=10 cursor=13:52ea6a34:::10:head 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=13 cursor=13:566253c9:::13:head 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=7 cursor=13:5c6b0b28:::7:head 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=14 cursor=13:62a1935d:::14:head 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=0 cursor=13:6cac518f:::0:head 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=15 cursor=13:863748b0:::15:head 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=11 cursor=13:89d3ae78:::11:head 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=5 cursor=13:b29083e3:::5:head 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=8 cursor=13:bd63b0f1:::8:head 2024-04-27T01:12:42.370 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=6 cursor=13:c4fdafeb:::6:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=3 cursor=13:cfc208b3:::3:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=4 cursor=13:d83876eb:::4:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=12 cursor=13:de5d7c5f:::12:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=9 cursor=13:e960b815:::9:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > oid=2 cursor=13:f905c69b:::2:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: FIRST> seek to MIN oid=1 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=1 cursor=13:02547ec2:::1:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=10 cursor=13:52ea6a34:::10:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=13 cursor=13:566253c9:::13:head 2024-04-27T01:12:42.371 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:566253c9:::13:head 2024-04-27T01:12:42.372 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-04-27T01:12:42.372 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=7 cursor=13:5c6b0b28:::7:head 2024-04-27T01:12:42.372 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-04-27T01:12:42.372 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-04-27T01:12:42.420 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=14 cursor=13:62a1935d:::14:head 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=0 cursor=13:6cac518f:::0:head 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=15 cursor=13:863748b0:::15:head 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:863748b0:::15:head 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=11 cursor=13:89d3ae78:::11:head 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=5 cursor=13:b29083e3:::5:head 2024-04-27T01:12:42.421 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=8 cursor=13:bd63b0f1:::8:head 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=6 cursor=13:c4fdafeb:::6:head 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=3 cursor=13:cfc208b3:::3:head 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=4 cursor=13:d83876eb:::4:head 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=12 cursor=13:de5d7c5f:::12:head 2024-04-27T01:12:42.422 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=9 cursor=13:e960b815:::9:head 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:e960b815:::9:head 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : oid=2 cursor=13:f905c69b:::2:head 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:d83876eb:::4:head expected=13:d83876eb:::4:head 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=4 expected=4 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:cfc208b3:::3:head expected=13:cfc208b3:::3:head 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-04-27T01:12:42.423 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=3 expected=3 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:f905c69b:::2:head expected=13:f905c69b:::2:head 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=2 expected=2 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:e960b815:::9:head 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:e960b815:::9:head expected=13:e960b815:::9:head 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=9 expected=9 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:de5d7c5f:::12:head expected=13:de5d7c5f:::12:head 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=12 expected=12 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:c4fdafeb:::6:head expected=13:c4fdafeb:::6:head 2024-04-27T01:12:42.424 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-04-27T01:12:42.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=6 expected=6 2024-04-27T01:12:42.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-04-27T01:12:42.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:bd63b0f1:::8:head expected=13:bd63b0f1:::8:head 2024-04-27T01:12:42.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-04-27T01:12:42.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=8 expected=8 2024-04-27T01:12:42.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-04-27T01:12:42.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:5c6b0b28:::7:head expected=13:5c6b0b28:::7:head 2024-04-27T01:12:42.425 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-04-27T01:12:42.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:12:42.614 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=7 expecte list_parallel: process_1_[75102]: starting. 2024-04-27T01:12:42.614 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[75102]: creating pool ceph_test_rados_list_parallel.smithi162-75069 2024-04-27T01:12:42.614 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[75102]: created object 0... 2024-04-27T01:12:42.614 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[75102]: created object 25... 2024-04-27T01:12:42.614 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[75102]: created object 49... 2024-04-27T01:12:42.614 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[75102]: finishing. 2024-04-27T01:12:42.614 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_1_[75102]: shutting down. 2024-04-27T01:12:42.614 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[75104]: starting. 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[75104]: listing objects. 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[75104]: listed object 0... 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[75104]: listed object 25... 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[75104]: saw 50 objects 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_2_[75104]: shutting down. 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[76317]: starting. 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[76317]: creating pool ceph_test_rados_list_parallel.smithi162-75069 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[76317]: created object 0... 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[76317]: created object 25... 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[76317]: created object 49... 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[76317]: finishing. 2024-04-27T01:12:42.615 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_3_[76317]: shutting down. 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[76318]: starting. 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[76318]: listing objects. 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[76318]: listed object 0... 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[76318]: listed object 25... 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[76318]: saw 43 objects 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_4_[76318]: shutting down. 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[76319]: starting. 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[76319]: removed 25 objects... 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[76319]: removed half of the objects 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[76319]: removed 50 objects... 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[76319]: removed 50 objects 2024-04-27T01:12:42.616 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_5_[76319]: shutting down. 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[76423]: starting. 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[76423]: creating pool ceph_test_rados_list_parallel.smithi162-75069 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[76423]: created object 0... 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[76423]: created object 25... 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[76423]: created object 49... 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[76423]: finishing. 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_6_[76423]: shutting down. 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[76424]: starting. 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[76424]: listing objects. 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[76424]: listed object 0... 2024-04-27T01:12:42.617 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[76424]: listed object 25... 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[76424]: listed object 50... 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[76424]: saw 56 objects 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_7_[76424]: shutting down. 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[76425]: starting. 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[76425]: added 25 objects... 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[76425]: added half of the objects 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[76425]: added 50 objects... 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[76425]: added 50 objects 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_8_[76425]: shutting down. 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.618 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.818 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[76471]: starting. 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[76471]: creating pool ceph_test_rados_list_parallel.smithi162-75069 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[76471]: created object 0... 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[76471]: created object 25... 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[76471]: created object 49... 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[76471]: finishing. 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_9_[76471]: shutting down. 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[76472]: starting. 2024-04-27T01:12:42.819 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[76472]: listing objects. 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[76472]: listed object 0... 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[76472]: listed object 25... 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[76472]: listed object 50... 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[76472]: listed object 75... 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[76472]: saw 97 objects 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_10_[76472]: shutting down. 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[76475]: starting. 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[76475]: removed 25 objects... 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[76475]: removed half of the objects 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[76475]: removed 50 objects... 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[76475]: removed 50 objects 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_13_[76475]: shutting down. 2024-04-27T01:12:42.820 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[76473]: starting. 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[76473]: added 25 objects... 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[76473]: added half of the objects 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[76473]: added 50 objects... 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[76473]: added 50 objects 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_11_[76473]: shutting down. 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[76474]: starting. 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[76474]: added 25 objects... 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[76474]: added half of the objects 2024-04-27T01:12:42.821 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[76474]: added 50 objects... 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[76474]: added 50 objects 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_12_[76474]: shutting down. 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[76553]: starting. 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[76553]: creating pool ceph_test_rados_list_parallel.smithi162-75069 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[76553]: created object 0... 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[76553]: created object 25... 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[76553]: created object 49... 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[76553]: finishing. 2024-04-27T01:12:42.822 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_14_[76553]: shutting down. 2024-04-27T01:12:42.823 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.823 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.823 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.823 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:42.823 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[76554]: starting. 2024-04-27T01:12:42.823 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[76554]: listing objects. 2024-04-27T01:12:42.823 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[76554]: listed object 0... 2024-04-27T01:12:43.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-74448-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-74719-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-74726-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-74772-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-74307-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.34694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-74405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-74551-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.34676 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-74351-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.34724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-74682-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-75054-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.34730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-74716-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75031-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.34769 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.34682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-74415-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74946-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-74739-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3527448613' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: osdmap e77: 8 total, 8 up, 8 in 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-74739-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:43.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:43.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-74448-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-74719-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-74726-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-74772-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-74307-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.34694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-74405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-74551-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.34676 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-74351-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.34724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-74682-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-75054-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.34730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-74716-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75031-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.34769 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.34682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-74415-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi162-74448-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi162-74719-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi162-74726-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi162-74772-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24667 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi162-74307-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.34694 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi162-74405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi162-74551-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.34676 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi162-74351-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.34724 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi162-74682-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24772 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi162-75054-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.34730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi162-74716-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75031-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.34769 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.34682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi162-74415-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74946-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-74739-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:43.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3527448613' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: osdmap e77: 8 total, 8 up, 8 in 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-74739-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24877 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74946-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-74739-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3527448613' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: osdmap e77: 8 total, 8 up, 8 in 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-74739-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:43.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:43.343 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[76554]: listed object 2 handler_error: Running main() from gmock_main.cc 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: handler_error: [----------] Global test environment set-up. 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (11074 ms) 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: handler_error: [----------] 1 test from neocls_handler_error (11074 ms total) 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: handler_error: 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: handler_error: [----------] Global test environment tear-down 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: handler_error: [==========] 1 test from 1 test suite ran. (11074 ms total) 2024-04-27T01:12:44.108 INFO:tasks.workunit.client.0.smithi162.stdout: handler_error: [ PASSED ] 1 test. 2024-04-27T01:12:44.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[37680]: pgmap v71: 1188 pgs: 44 creating+peering, 132 active+clean, 1012 unknown; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2024-04-27T01:12:44.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:44.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[37680]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi162-75471-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:44.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[37680]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi162-75564-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi162-75564-1"}]': finished 2024-04-27T01:12:44.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[37680]: osdmap e78: 8 total, 8 up, 8 in 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[37680]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[37680]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[26714]: pgmap v71: 1188 pgs: 44 creating+peering, 132 active+clean, 1012 unknown; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[26714]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi162-75471-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[26714]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi162-75564-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi162-75564-1"}]': finished 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:44.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[26714]: osdmap e78: 8 total, 8 up, 8 in 2024-04-27T01:12:44.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:44.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[26714]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:44.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:44.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:44 smithi162 ceph-mon[26714]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:44 smithi181 ceph-mon[30237]: pgmap v71: 1188 pgs: 44 creating+peering, 132 active+clean, 1012 unknown; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2024-04-27T01:12:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:44 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:44 smithi181 ceph-mon[30237]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi162-75471-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:44 smithi181 ceph-mon[30237]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi162-75564-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi162-75564-1"}]': finished 2024-04-27T01:12:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:44 smithi181 ceph-mon[30237]: osdmap e78: 8 total, 8 up, 8 in 2024-04-27T01:12:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:44 smithi181 ceph-mon[30237]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:44 smithi181 ceph-mon[30237]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.160 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: Running main() from gmock_main.cc 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [----------] Global test environment set-up. 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (76 ms) 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (51 ms) 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (11166 ms) 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:42.924342+0000 mon.a [INF] from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi162-75471-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:45.161 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:42.924433+0000 mon.a [INF] from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi162-75564-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi162-75564-1"}]': finished 2024-04-27T01:12:45.162 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:43.215025+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.162 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:43.816850+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:45.162 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:43.817489+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:45.162 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:43.825683+0000 mon.a [INF] from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.162 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.087331+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:45.162 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.103496+0000 mon.a [INF] from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-74739-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]': finished 2024-04-27T01:12:45.162 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.103650+0000 mon.a [INF] from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:45.162 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.118641+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.162 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.122989+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:45.162 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.123791+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/1679646429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: pgmap v74: 1172 pgs: 99 creating+peering, 964 active+clean, 109 unknown; 611 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 15 KiB/s wr, 20 op/s 2024-04-27T01:12:45.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:45.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-74739-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]': finished 2024-04-27T01:12:45.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:45.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: osdmap e79: 8 total, 8 up, 8 in 2024-04-27T01:12:45.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:45.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1679646429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:12:45.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:12:45.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:12:45.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:12:45.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:12:45.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:12:45.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:12:45.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:12:45.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: pgmap v74: 1172 pgs: 99 creating+peering, 964 active+clean, 109 unknown; 611 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 15 KiB/s wr, 20 op/s 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-74739-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]': finished 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: osdmap e79: 8 total, 8 up, 8 in 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1679646429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-27T01:12:45.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:12:45.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:12:45.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:12:45.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: pgmap v74: 1172 pgs: 99 creating+peering, 964 active+clean, 109 unknown; 611 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 15 KiB/s wr, 20 op/s 2024-04-27T01:12:45.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:45.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi162-74739-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]': finished 2024-04-27T01:12:45.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:45.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: osdmap e79: 8 total, 8 up, 8 in 2024-04-27T01:12:45.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:45.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1679646429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:12:45.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:12:45.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:12:45.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:12:45.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:12:45.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:12:45.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:12:46.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.130534+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:46.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.144293+0000 mon.a [INF] from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:46.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.194040+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.201462+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.203599+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.448524+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.448923+0000 mon.a [INF] from='client.35219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.449187+0000 mon.a [INF] from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.592955+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.593604+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.594037+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.594417+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.594789+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.595238+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.595728+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.596076+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.596449+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.596798+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:12:46.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:44.201803+0000 client.admin [INF] onexx 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.170824+0000 mon.a [INF] from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.181403+0000 mon.a [INF] from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.181467+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.181516+0000 mon.a [INF] from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.181567+0000 mon.a [INF] from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.187827+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.189301+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.191117+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.191417+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.196587+0000 mon.a [INF] from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.197168+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.197563+0000 mon.a [INF] from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.197850+0000 mon.a [INF] from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.198614+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.240687+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.454725+0000 mon.a [INF] from='cl api_cmd_pp: Running main() from gmock_main.cc 2024-04-27T01:12:46.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-04-27T01:12:46.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (80 ms) 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (68 ms) 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (13475 ms) 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (13623 ms total) 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (13623 ms total) 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout:5... 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[76554]: listed object 50... 2024-04-27T01:12:46.496 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[76554]: listed object 75... 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[76554]: listed object 100... 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[76554]: listed object 125... 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[76554]: saw 150 objects 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_15_[76554]: shutting down. 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[76555]: starting. 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[76555]: added 25 objects... 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[76555]: added half of the objects 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[76555]: added 50 objects... 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[76555]: added 50 objects 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: process_16_[76555]: shutting down. 2024-04-27T01:12:46.497 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:46.498 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:46.498 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:46.498 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:46.498 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******************************* 2024-04-27T01:12:46.498 INFO:tasks.workunit.client.0.smithi162.stdout: list_parallel: ******* SUCCESS ********** 2024-04-27T01:12:46.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-04-27T01:12:46.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-04-27T01:12:46.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-04-27T01:12:46.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-04-27T01:12:46.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (13563 ms) 2024-04-27T01:12:46.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (13563 ms total) 2024-04-27T01:12:46.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_cls_remote_reads: 2024-04-27T01:12:46.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-04-27T01:12:46.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (13563 ms total) 2024-04-27T01:12:46.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-04-27T01:12:46.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-04-27T01:12:46.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (386 ms) 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (11 ms) 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (10 ms) 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (25 ms) 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-04-27T01:12:46.502 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (8 ms) 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (5 ms) 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (8 ms) 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (4 ms) 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (7 ms) 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-04-27T01:12:46.503 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (7 ms) 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (14 ms) 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (12 ms) 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (9 ms) 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (6 ms) 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (525 ms total) 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (13593 ms total) 2024-04-27T01:12:46.504 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-04-27T01:12:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: onexx 2024-04-27T01:12:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.6 deep-scrub starts 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.6 deep-scrub ok 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.3 deep-scrub starts 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.0 deep-scrub starts 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.0 deep-scrub ok 2024-04-27T01:12:46.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.3 deep-scrub ok 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: pgmap v76: 1012 pgs: 73 creating+peering, 740 active+clean, 199 unknown; 610 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: osdmap e80: 8 total, 8 up, 8 in 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:46.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.2 deep-scrub starts 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.2 deep-scrub ok 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.1 deep-scrub starts 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.1 deep-scrub ok 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.5 deep-scrub starts 2024-04-27T01:12:46.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: 20.5 deep-scrub ok 2024-04-27T01:12:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-27T01:12:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.514 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:46 smithi181 ceph-mon[30237]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: onexx 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:12:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.6 deep-scrub starts 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.6 deep-scrub ok 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.3 deep-scrub starts 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.0 deep-scrub starts 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.0 deep-scrub ok 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.3 deep-scrub ok 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: pgmap v76: 1012 pgs: 73 creating+peering, 740 active+clean, 199 unknown; 610 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:46.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: osdmap e80: 8 total, 8 up, 8 in 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.2 deep-scrub starts 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.2 deep-scrub ok 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.1 deep-scrub starts 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.1 deep-scrub ok 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.5 deep-scrub starts 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: 20.5 deep-scrub ok 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[26714]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: onexx 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:12:46.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.6 deep-scrub starts 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.6 deep-scrub ok 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.3 deep-scrub starts 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.0 deep-scrub starts 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.0 deep-scrub ok 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.3 deep-scrub ok 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: pgmap v76: 1012 pgs: 73 creating+peering, 740 active+clean, 199 unknown; 610 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi162-74307-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/451103966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:46.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: osdmap e80: 8 total, 8 up, 8 in 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.2 deep-scrub starts 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.2 deep-scrub ok 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.1 deep-scrub starts 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.1 deep-scrub ok 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.5 deep-scrub starts 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: 20.5 deep-scrub ok 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:46.596 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:46 smithi162 ceph-mon[37680]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:12:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-27T01:12:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-27T01:12:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: twoxx 2024-04-27T01:12:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-27T01:12:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-74351-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-74716-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-74415-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-74325-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi162-75471-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: osdmap e81: 8 total, 8 up, 8 in 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3503593114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/373625349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-74351-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-74716-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:47.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:47.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-74415-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:47.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-74325-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-27T01:12:47.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:47.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:47.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:47.513 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:47 smithi181 ceph-mon[30237]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi162-75471-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:12:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-27T01:12:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-27T01:12:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: twoxx 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-74351-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-74716-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-74415-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-74325-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi162-75471-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: osdmap e81: 8 total, 8 up, 8 in 2024-04-27T01:12:47.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3503593114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:47.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/373625349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:12:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-27T01:12:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-27T01:12:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: twoxx 2024-04-27T01:12:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-27T01:12:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi162-75471-1"}]': finished 2024-04-27T01:12:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi162-74325-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:47.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-74351-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-74716-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-74415-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/464708297' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-74325-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-27T01:12:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi162-75471-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: osdmap e81: 8 total, 8 up, 8 in 2024-04-27T01:12:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3503593114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:47.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/373625349' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:47.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-74351-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:47.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-74716-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:47.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:47.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-74415-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:47.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-74325-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-27T01:12:47.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:47.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:47.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:47.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[37680]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi162-75471-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:47.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-74351-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:47.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-74716-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:47.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:47.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-74415-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:47.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.35081 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-74325-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-27T01:12:47.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:47.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:47.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:47.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:47 smithi162 ceph-mon[26714]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi162-75471-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:48.478 INFO:tasks.workunit.client.0.smithi162.stdout: cls: Running main() from gmock_main.cc 2024-04-27T01:12:48.478 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-04-27T01:12:48.478 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [----------] Global test environment set-up. 2024-04-27T01:12:48.478 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-04-27T01:12:48.478 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-04-27T01:12:48.478 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [ OK ] NeoRadosCls.DNE (11109 ms) 2024-04-27T01:12:48.478 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-04-27T01:12:48.478 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (4367 ms) 2024-04-27T01:12:48.479 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [----------] 2 tests from NeoRadosCls (15476 ms total) 2024-04-27T01:12:48.479 INFO:tasks.workunit.client.0.smithi162.stdout: cls: 2024-04-27T01:12:48.479 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [----------] Global test environment tear-down 2024-04-27T01:12:48.479 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [==========] 2 tests from 1 test suite ran. (15476 ms total) 2024-04-27T01:12:48.479 INFO:tasks.workunit.client.0.smithi162.stdout: cls: [ PASSED ] 2 tests. 2024-04-27T01:12:48.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: 20.4 deep-scrub starts 2024-04-27T01:12:48.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: 20.4 deep-scrub ok 2024-04-27T01:12:48.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: pgmap v79: 756 pgs: 128 creating+peering, 628 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 36 MiB/s wr, 506 op/s 2024-04-27T01:12:48.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-27T01:12:48.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]': finished 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-74325-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi162-75471-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: osdmap e82: 8 total, 8 up, 8 in 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:48.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:48 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:48.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: 20.4 deep-scrub starts 2024-04-27T01:12:48.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: 20.4 deep-scrub ok 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: pgmap v79: 756 pgs: 128 creating+peering, 628 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 36 MiB/s wr, 506 op/s 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]': finished 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-74325-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi162-75471-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: osdmap e82: 8 total, 8 up, 8 in 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:12:48.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: 20.4 deep-scrub starts 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: 20.4 deep-scrub ok 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: pgmap v79: 756 pgs: 128 creating+peering, 628 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 36 MiB/s wr, 506 op/s 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]': finished 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-74325-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi162-75471-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:48.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: osdmap e82: 8 total, 8 up, 8 in 2024-04-27T01:12:48.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:48.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:48.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:48.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:12:48.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:48.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:48 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:12:49.228 INFO:tasks.workunit.client.0.smithi162.stdout:ient.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.455436+0000 mon.a [INF] from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.455957+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.458064+0000 mon.a [INF] from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.458401+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:45.458778+0000 mon.a [INF] from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi162-74415-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:46.164694+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:46.164911+0000 mon.a [INF] from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:46.165635+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:46.165845+0000 mon.a [INF] from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:46.166365+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:46.166741+0000 mon.a [INF] from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi162-74351-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:49.229 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:47.236218+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-27T01:12:49.230 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:47.242466+0000 mon.a [INF] from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi162-74360-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi162-74360-22"}]': finished 2024-04-27T01:12:49.230 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:47.242524+0000 mon.a [INF] from='client.35081 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi162-74325-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-27T01:12:49.230 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:47.242570+0000 mon.a [INF] from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi162-75471-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:49.795 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:47.251566+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/32 api_io_pp: Running main() from gmock_main.cc 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] Global test environment set-up. 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: seed 74360 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (60 ms) 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (11 ms) 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (2 ms) 2024-04-27T01:12:49.796 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (5 ms) 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2 ms) 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2 ms) 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (4 ms) 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (2 ms) 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3 ms) 2024-04-27T01:12:49.797 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (3 ms) 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (2 ms) 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (2 ms) 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3 ms) 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (6 ms) 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (3 ms) 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (3 ms) 2024-04-27T01:12:49.798 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (1 ms) 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (2 ms) 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (119 ms total) 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (2526 ms) 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (13 ms) 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-04-27T01:12:49.799 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (7 ms) 2024-04-27T01:12:49.800 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-04-27T01:12:49.800 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (4 ms) 2024-04-27T01:12:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:49 smithi181 ceph-mon[30237]: 20.8 deep-scrub starts 2024-04-27T01:12:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:49 smithi181 ceph-mon[30237]: 20.8 deep-scrub ok 2024-04-27T01:12:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:49 smithi181 ceph-mon[30237]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:49 smithi181 ceph-mon[30237]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:49 smithi181 ceph-mon[30237]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:49 smithi181 ceph-mon[30237]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:50.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:12:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:12:50.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[37680]: 20.8 deep-scrub starts 2024-04-27T01:12:50.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[37680]: 20.8 deep-scrub ok 2024-04-27T01:12:50.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[37680]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:50.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[37680]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:50.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[37680]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:50.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[37680]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[26714]: 20.8 deep-scrub starts 2024-04-27T01:12:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[26714]: 20.8 deep-scrub ok 2024-04-27T01:12:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[26714]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[26714]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[26714]: from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:49 smithi162 ceph-mon[26714]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:50.690 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] L17499835' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:50.690 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.218374+0000 mon.a [INF] from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:50.690 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.219300+0000 mon.a [INF] from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:50.690 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.219836+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:50.690 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.224148+0000 mon.a [INF] from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:12:50.691 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.224559+0000 mon.a [INF] from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:50.691 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.467662+0000 mon.a [INF] from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:50.691 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.473963+0000 mon.a [INF] from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:50.691 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.474105+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:50.691 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.474321+0000 mon.a [INF] from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:50.691 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.475241+0000 mon.a [INF] from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-74351-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-74351-16"}]': finished 2024-04-27T01:12:50.691 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.475298+0000 mon.a [INF] from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-74716-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-74716-7"}]': finished 2024-04-27T01:12:50.691 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.475344+0000 mon.a [INF] from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-74415-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-74415-10"}]': finished 2024-04-27T01:12:50.691 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.475390+0000 mon.a [INF] from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:50.691 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.481255+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:51.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: 20.7 deep-scrub starts 2024-04-27T01:12:51.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: 20.7 deep-scrub ok 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-74351-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-74351-16"}]': finished 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-74716-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-74716-7"}]': finished 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-74415-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-74415-10"}]': finished 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: pgmap v82: 780 pgs: 192 unknown, 128 creating+peering, 460 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 527 op/s 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-27T01:12:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: threexx 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: osdmap e83: 8 total, 8 up, 8 in 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]': finished 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: osdmap e84: 8 total, 8 up, 8 in 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:51.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:50 smithi181 ceph-mon[30237]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: 20.7 deep-scrub starts 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: 20.7 deep-scrub ok 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-74351-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-74351-16"}]': finished 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-74716-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-74716-7"}]': finished 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-74415-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-74415-10"}]': finished 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:51.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: pgmap v82: 780 pgs: 192 unknown, 128 creating+peering, 460 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 527 op/s 2024-04-27T01:12:51.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-27T01:12:51.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: threexx 2024-04-27T01:12:51.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:51.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: osdmap e83: 8 total, 8 up, 8 in 2024-04-27T01:12:51.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:51.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:51.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]': finished 2024-04-27T01:12:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: 20.7 deep-scrub starts 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: 20.7 deep-scrub ok 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi162-74351-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi162-74351-16"}]': finished 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi162-74716-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi162-74716-7"}]': finished 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi162-74415-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi162-74415-10"}]': finished 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: pgmap v82: 780 pgs: 192 unknown, 128 creating+peering, 460 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 527 op/s 2024-04-27T01:12:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: threexx 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: osdmap e83: 8 total, 8 up, 8 in 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]': finished 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: osdmap e84: 8 total, 8 up, 8 in 2024-04-27T01:12:51.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[37680]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: osdmap e84: 8 total, 8 up, 8 in 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:51.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:50 smithi162 ceph-mon[26714]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.484096+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551- cmd: Running main() from gmock_main.cc 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [----------] Global test environment set-up. 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (9263 ms) 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (3888 ms) 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (5807 ms) 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [----------] 3 tests from NeoRadosCmd (18958 ms total) 2024-04-27T01:12:51.999 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: 2024-04-27T01:12:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [----------] Global test environment tear-down 2024-04-27T01:12:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [==========] 3 tests from 1 test suite ran. (18959 ms total) 2024-04-27T01:12:52.000 INFO:tasks.workunit.client.0.smithi162.stdout: cmd: [ PASSED ] 3 tests. 2024-04-27T01:12:52.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-27T01:12:52.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-74772-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:52.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:52.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: fourxx 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]': finished 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]': finished 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]': finished 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: osdmap e85: 8 total, 8 up, 8 in 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-4", "overlaypool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-74448-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2093376100' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-74307-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-4", "overlaypool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-74448-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.35375 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-74307-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-74772-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:52.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-27T01:12:52.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-74772-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:52.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: fourxx 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]': finished 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]': finished 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]': finished 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: osdmap e85: 8 total, 8 up, 8 in 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-4", "overlaypool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-74448-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:52.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2093376100' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-74307-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:52.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-4", "overlaypool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:52.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:52.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-27T01:12:52.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-74772-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:52.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:52.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-27T01:12:52.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-27T01:12:52.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: fourxx 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]': finished 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]': finished 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]': finished 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: osdmap e85: 8 total, 8 up, 8 in 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-4", "overlaypool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/340461374' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-74448-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2093376100' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-74307-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:52.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-4", "overlaypool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:52.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-74448-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.35375 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-74307-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-74772-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-27T01:12:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-27T01:12:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-74448-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.35375 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-74307-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-74772-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.25243 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.24736 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:52.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:52.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:12:52.766 INFO:tasks.workunit.client.0.smithi162.stdout:ibRadosIoECPP.OverlappingWriteRoundTripPP 2024-04-27T01:12:52.767 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (6 ms) 2024-04-27T01:12:52.767 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-04-27T01:12:52.767 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6 ms) 2024-04-27T01:12:52.767 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-04-27T01:12:52.767 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-04-27T01:12:52.767 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-04-27T01:12:52.767 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-04-27T01:12:52.767 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (6 ms) 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2 ms) 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9 ms) 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (9 ms) 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (3 ms) 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-04-27T01:12:52.768 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (6 ms) 2024-04-27T01:12:52.769 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (2620 ms total) 2024-04-27T01:12:52.769 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: 2024-04-27T01:12:52.769 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [----------] Global test environment tear-down 2024-04-27T01:12:52.769 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (20094 ms total) 2024-04-27T01:12:52.769 INFO:tasks.workunit.client.0.smithi162.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-04-27T01:12:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: pgmap v85: 900 pgs: 384 unknown, 64 creating+peering, 452 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:12:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: 20.9 deep-scrub starts 2024-04-27T01:12:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: 20.9 deep-scrub ok 2024-04-27T01:12:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-27T01:12:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-4", "overlaypool": "test-rados-api-smithi162-74551-4-cache"}]': finished 2024-04-27T01:12:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi162-75564-1"}]': finished 2024-04-27T01:12:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35375 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-74307-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]': finished 2024-04-27T01:12:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-74772-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:53.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]': finished 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-4-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2"}]: dispatch 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-74772-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi162-75564-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: osdmap e86: 8 total, 8 up, 8 in 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3948427981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:53.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-4-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-74772-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi162-75564-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35393 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:53.013 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:52 smithi181 ceph-mon[30237]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:53.031 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[75240]: starting. 2024-04-27T01:12:53.031 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[75240]: creating pool ceph_test_rados_delete_pools_parallel.smithi162-75173 2024-04-27T01:12:53.031 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[75240]: created object 0... 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[75240]: created object 25... 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[75240]: created object 49... 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[75240]: finishing. 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_1_[75240]: shutting down. 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_2_[75241]: starting. 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_2_[75241]: deleting pool ceph_test_rados_delete_pools_parallel.smithi162-75173 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_2_[75241]: shutting down. 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******************************* 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[76853]: starting. 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[76853]: creating pool ceph_test_rados_delete_pools_parallel.smithi162-75173 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[76853]: created object 0... 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[76853]: created object 25... 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[76853]: created object 49... 2024-04-27T01:12:53.032 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[76853]: finishing. 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_3_[76853]: shutting down. 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******************************* 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[76855]: starting. 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[76855]: listing objects. 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[76855]: listed object 0... 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[76855]: listed object 25... 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[76855]: saw 50 objects 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_5_[76855]: shutting down. 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******************************* 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_4_[76854]: starting. 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_4_[76854]: deleting pool ceph_test_rados_delete_pools_parallel.smithi162-75173 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: process_4_[76854]: shutting down. 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******************************* 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******************************* 2024-04-27T01:12:53.033 INFO:tasks.workunit.client.0.smithi162.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[75127]: starting. 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[75127]: creating pool ceph_test_rados_open_pools_parallel.smithi162-75091 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[75127]: created object 0... 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[75127]: created object 25... 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[75127]: created object 49... 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[75127]: finishing. 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_1_[75127]: shutting down. 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_2_[75128]: starting. 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_2_[75128]: rados_pool_create. 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_2_[75128]: rados_ioctx_create. 2024-04-27T01:12:53.034 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_2_[75128]: shutting down. 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: ******************************* 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[76856]: starting. 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[76856]: creating pool ceph_test_rados_open_pools_parallel.smithi162-75091 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[76856]: created object 0... 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[76856]: created object 25... 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[76856]: created object 49... 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[76856]: finishing. 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_3_[76856]: shutting down. 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: ******************************* 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_4_[76857]: starting. 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_4_[76857]: rados_pool_create. 2024-04-27T01:12:53.035 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_4_[76857]: rados_ioctx_create. 2024-04-27T01:12:53.036 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: process_4_[76857]: shutting down. 2024-04-27T01:12:53.036 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: ******************************* 2024-04-27T01:12:53.036 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: ******************************* 2024-04-27T01:12:53.036 INFO:tasks.workunit.client.0.smithi162.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-04-27T01:12:53.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: pgmap v85: 900 pgs: 384 unknown, 64 creating+peering, 452 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:12:53.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: 20.9 deep-scrub starts 2024-04-27T01:12:53.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: 20.9 deep-scrub ok 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-4", "overlaypool": "test-rados-api-smithi162-74551-4-cache"}]': finished 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi162-75564-1"}]': finished 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35375 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-74307-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]': finished 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-74772-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]': finished 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-4-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2"}]: dispatch 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-74772-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi162-75564-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:53.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: osdmap e86: 8 total, 8 up, 8 in 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3948427981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-4-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-74772-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi162-75564-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35393 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:53.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:53.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:53.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[37680]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:53.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: pgmap v85: 900 pgs: 384 unknown, 64 creating+peering, 452 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:12:53.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: 20.9 deep-scrub starts 2024-04-27T01:12:53.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: 20.9 deep-scrub ok 2024-04-27T01:12:53.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.25243 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-27T01:12:53.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-4", "overlaypool": "test-rados-api-smithi162-74551-4-cache"}]': finished 2024-04-27T01:12:53.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi162-75564-1"}]': finished 2024-04-27T01:12:53.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35375 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi162-74307-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi162-74360-22"}]': finished 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi162-74772-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.24736 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi162-74739-1"}]': finished 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-4-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1801521952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2"}]: dispatch 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-74772-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi162-75564-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: osdmap e86: 8 total, 8 up, 8 in 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3948427981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-4-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-74772-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi162-75564-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35393 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:53.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:52 smithi162 ceph-mon[26714]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:53.535 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: Running main() from gmock_main.cc 2024-04-27T01:12:53.535 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-04-27T01:12:53.535 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [----------] Global test environment set-up. 2024-04-27T01:12:53.535 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-04-27T01:12:53.536 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-04-27T01:12:53.536 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (13149 ms) 2024-04-27T01:12:53.536 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-04-27T01:12:53.536 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (7347 ms) 2024-04-27T01:12:53.536 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (20496 ms total) 2024-04-27T01:12:53.536 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: 2024-04-27T01:12:53.536 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [----------] Global test environment tear-down 2024-04-27T01:12:53.536 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (20496 ms total) 2024-04-27T01:12:53.536 INFO:tasks.workunit.client.0.smithi162.stdout: ec_io: [ PASSED ] 2 tests. 2024-04-27T01:12:53.539 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-04-27T01:12:53.539 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-04-27T01:12:53.539 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-04-27T01:12:53.539 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-04-27T01:12:53.539 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: seed 74716 2024-04-27T01:12:53.539 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (293 ms) 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (76 ms) 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (3 ms) 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (19 ms) 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (391 ms total) 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-04-27T01:12:53.540 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (2370 ms) 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (4 ms) 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (2 ms) 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (7 ms) 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (2383 ms total) 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (20741 ms total) 2024-04-27T01:12:53.541 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: Running main() from gmock_main.cc 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] Global test environment set-up. 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (346 ms) 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: no timeout :/ 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: no timeout :/ 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: no timeout :/ 2024-04-27T01:12:53.542 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: no timeout :/ 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: no timeout :/ 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (52 ms) 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (10 ms) 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.Checksum (5 ms) 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (6 ms) 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (5 ms) 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (6 ms) 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (3 ms) 2024-04-27T01:12:53.543 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (5 ms) 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (5 ms) 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.RmXattr (14 ms) 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIo.XattrIter (10 ms) 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] 14 tests from LibRadosIo (471 ms total) 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-04-27T01:12:53.544 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (2372 ms) 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (7 ms) 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (5 ms) 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (5 ms) 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (6 ms) 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (2 ms) 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-04-27T01:12:53.545 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-04-27T01:12:53.546 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-04-27T01:12:53.546 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (8 ms) 2024-04-27T01:12:53.546 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-04-27T01:12:53.546 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (3 ms) 2024-04-27T01:12:53.546 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] 10 tests from LibRadosIoEC (2418 ms total) 2024-04-27T01:12:53.546 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: 2024-04-27T01:12:53.546 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [----------] Global test environment tear-down 2024-04-27T01:12:53.546 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [==========] 24 tests from 2 test suites ran. (20869 ms total) 2024-04-27T01:12:53.546 INFO:tasks.workunit.client.0.smithi162.stdout: api_io: [ PASSED ] 24 tests. 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout:4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.752072+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.752349+0000 mon.c [INF] from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:48.987427+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:49.240615+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/1754732792' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:49.240757+0000 client.admin [INF] threexx 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:49.696439+0000 mon.a [INF] from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi162-75564-1"}]: dispatch 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:49.697016+0000 mon.a [INF] from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:49.703294+0000 mon.a [INF] from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi162-75471-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi162-75471-2"}]': finished 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:49.703369+0000 mon.a [INF] from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:49.703421+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:53.696 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:49.709567+0000 mon.b [INF] from='client.? v1:172.21.15.162:0/3845834920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi162-74739-1"}]: dispatch 2024-04-27T01:12:53.697 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:50.458879+0000 mon.a [INF] from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:53.697 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:50.459409+0000 mon.a [INF] from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:53.697 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: got: 2024-04-27T01:12:50.697929+0000 mon.a [INF] from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:53.697 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (9555 ms) 2024-04-27T01:12:53.697 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (20848 ms total) 2024-04-27T01:12:53.697 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: 2024-04-27T01:12:53.697 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [----------] Global test environment tear-down 2024-04-27T01:12:53.697 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (20848 ms total) 2024-04-27T01:12:53.697 INFO:tasks.workunit.client.0.smithi162.stdout: api_cmd: [ PASSED ] 4 tests. 2024-04-27T01:12:54.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:54.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-74448-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-74448-10"}]': finished 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-4-cache", "mode": "writeback"}]': finished 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]': finished 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi162-75564-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35393 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi162-75471-2"}]': finished 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]': finished 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]': finished 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi162-75564-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:54.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:54.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-27T01:12:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:12:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-74448-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-74448-10"}]': finished 2024-04-27T01:12:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-4-cache", "mode": "writeback"}]': finished 2024-04-27T01:12:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]': finished 2024-04-27T01:12:54.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:54.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi162-75564-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:54.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35393 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:54.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi162-75471-2"}]': finished 2024-04-27T01:12:54.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]': finished 2024-04-27T01:12:54.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]': finished 2024-04-27T01:12:54.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:54.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:54.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi162-75564-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:54.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:54.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-4"}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: osdmap e87: 8 total, 8 up, 8 in 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi162-75564-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/407226464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-4"}]: dispatch 2024-04-27T01:12:54.092 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[37680]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-74772-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-74772-7"}]': finished 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-4"}]: dispatch 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: osdmap e87: 8 total, 8 up, 8 in 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi162-75564-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/407226464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:54.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:12:54.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-4"}]: dispatch 2024-04-27T01:12:54.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:54.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:53 smithi162 ceph-mon[26714]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-74772-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-74772-7"}]': finished 2024-04-27T01:12:54.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi162-74448-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi162-74448-10"}]': finished 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-4-cache", "mode": "writeback"}]': finished 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi162-74360-22"}]': finished 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi162-74536-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi162-75564-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35393 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:54.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi162-75471-2"}]': finished 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi162-74716-7"}]': finished 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi162-74351-16"}]': finished 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/318377568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/180893121' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi162-75564-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3217499835' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-4"}]: dispatch 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: osdmap e87: 8 total, 8 up, 8 in 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:54.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35255 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]: dispatch 2024-04-27T01:12:54.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.25246 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]: dispatch 2024-04-27T01:12:54.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi162-75564-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:54.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi162-75471-2"}]: dispatch 2024-04-27T01:12:54.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/407226464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:54.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:12:54.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-4"}]: dispatch 2024-04-27T01:12:54.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:54.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:53 smithi181 ceph-mon[30237]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi162-74772-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi162-74772-7"}]': finished 2024-04-27T01:12:55.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]': finished 2024-04-27T01:12:55.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]': finished 2024-04-27T01:12:55.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi162-75471-2"}]': finished 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-4"}]': finished 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74739-6", "pg_num": 4}]: dispatch 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: osdmap e88: 8 total, 8 up, 8 in 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:55.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.25381 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:55.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:55.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:12:55.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]': finished 2024-04-27T01:12:55.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]': finished 2024-04-27T01:12:55.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi162-75471-2"}]': finished 2024-04-27T01:12:55.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-27T01:12:55.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-4"}]': finished 2024-04-27T01:12:55.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:55.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:55.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:55.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:12:55.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74739-6", "pg_num": 4}]: dispatch 2024-04-27T01:12:55.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: osdmap e88: 8 total, 8 up, 8 in 2024-04-27T01:12:55.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:55.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:55.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:12:55.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:55.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.25381 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:55.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:55.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:54 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:12:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.35255 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi162-74716-7"}]': finished 2024-04-27T01:12:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.25246 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi162-74351-16"}]': finished 2024-04-27T01:12:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi162-75471-2"}]': finished 2024-04-27T01:12:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-27T01:12:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-4"}]': finished 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3615256918' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74739-6", "pg_num": 4}]: dispatch 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: osdmap e88: 8 total, 8 up, 8 in 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]: dispatch 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.25381 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:55.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:54 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: Running main() from gmock_main.cc 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: [----------] Global test environment set-up. 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: [----------] 3 tests from NeoradosList 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: [ OK ] NeoradosList.ListObjects (13161 ms) 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: [ OK ] NeoradosList.ListObjectsNS (5784 ms) 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3612 ms) 2024-04-27T01:12:55.628 INFO:tasks.workunit.client.0.smithi162.stdout: list: [----------] 3 tests from NeoradosList (22557 ms total) 2024-04-27T01:12:55.629 INFO:tasks.workunit.client.0.smithi162.stdout: list: 2024-04-27T01:12:55.629 INFO:tasks.workunit.client.0.smithi162.stdout: list: [----------] Global test environment tear-down 2024-04-27T01:12:55.629 INFO:tasks.workunit.client.0.smithi162.stdout: list: [==========] 3 tests from 1 test suite ran. (22557 ms total) 2024-04-27T01:12:55.629 INFO:tasks.workunit.client.0.smithi162.stdout: list: [ PASSED ] 3 tests. 2024-04-27T01:12:55.948 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: Running main() from gmock_main.cc 2024-04-27T01:12:55.948 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-04-27T01:12:55.948 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [----------] Global test environment set-up. 2024-04-27T01:12:55.948 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-04-27T01:12:55.948 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-04-27T01:12:55.948 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: handle_notify cookie 94447392293040 notify_id 330712481792 notifier_gid 24925 2024-04-27T01:12:55.948 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (13396 ms) 2024-04-27T01:12:55.948 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-04-27T01:12:55.948 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: Trying... 2024-04-27T01:12:55.948 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: handle_notify cookie 94447392537968 notify_id 365072220160 notifier_gid 25276 2024-04-27T01:12:55.949 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: Waiting for 3.000000000s 2024-04-27T01:12:55.949 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: Timed out. 2024-04-27T01:12:55.949 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: Flushing... 2024-04-27T01:12:55.949 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: Flushed... 2024-04-27T01:12:55.949 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (9455 ms) 2024-04-27T01:12:55.949 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (22851 ms total) 2024-04-27T01:12:55.949 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: 2024-04-27T01:12:55.949 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [----------] Global test environment tear-down 2024-04-27T01:12:55.949 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (22851 ms total) 2024-04-27T01:12:55.949 INFO:tasks.workunit.client.0.smithi162.stdout: watch_notify: [ PASSED ] 2 tests. 2024-04-27T01:12:56.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: pgmap v88: 692 pgs: 64 creating+peering, 104 unknown, 524 active+clean; 145 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-04-27T01:12:56.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi162-75564-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi162-75564-2"}]': finished 2024-04-27T01:12:56.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]': finished 2024-04-27T01:12:56.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-27T01:12:56.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:56.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.25381 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:56.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-74719-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:56.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:12:56.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: osdmap e89: 8 total, 8 up, 8 in 2024-04-27T01:12:56.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74739-6", "pg_num": 4}]: dispatch 2024-04-27T01:12:56.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:56.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:12:56.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:56.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-74719-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:56.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:55 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74739-6", "pg_num": 4}]': finished 2024-04-27T01:12:56.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: pgmap v88: 692 pgs: 64 creating+peering, 104 unknown, 524 active+clean; 145 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-04-27T01:12:56.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi162-75564-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi162-75564-2"}]': finished 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]': finished 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.25381 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-74719-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: osdmap e89: 8 total, 8 up, 8 in 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74739-6", "pg_num": 4}]: dispatch 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-74719-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:56.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74739-6", "pg_num": 4}]': finished 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: pgmap v88: 692 pgs: 64 creating+peering, 104 unknown, 524 active+clean; 145 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi162-75564-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi162-75564-2"}]': finished 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.35087 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-4", "tierpool": "test-rados-api-smithi162-74551-4-cache"}]': finished 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.25381 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-74719-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1288737803' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: osdmap e89: 8 total, 8 up, 8 in 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74739-6", "pg_num": 4}]: dispatch 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:56.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:12:56.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:56.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-74719-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:12:56.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:55 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74739-6", "pg_num": 4}]': finished 2024-04-27T01:12:56.650 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: Running main() from gmock_main.cc 2024-04-27T01:12:56.650 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-04-27T01:12:56.650 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] Global test environment set-up. 2024-04-27T01:12:56.650 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-04-27T01:12:56.650 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (329 ms) 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.LockShared (4 ms) 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1022 ms) 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1005 ms) 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (4 ms) 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.Unlock (16 ms) 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (27 ms) 2024-04-27T01:12:56.651 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (20 ms) 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] 8 tests from LibRadosLock (2427 ms total) 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (2365 ms) 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (6 ms) 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1091 ms) 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1017 ms) 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3 ms) 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-04-27T01:12:56.652 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-04-27T01:12:56.653 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-04-27T01:12:56.653 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (4 ms) 2024-04-27T01:12:56.653 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-04-27T01:12:56.653 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (2 ms) 2024-04-27T01:12:56.653 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (4492 ms total) 2024-04-27T01:12:56.653 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: 2024-04-27T01:12:56.653 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [----------] Global test environment tear-down 2024-04-27T01:12:56.653 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (23956 ms total) 2024-04-27T01:12:56.653 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock: [ PASSED ] 16 tests. 2024-04-27T01:12:58.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: pgmap v91: 796 pgs: 280 unknown, 516 active+clean; 145 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 5.2 KiB/s wr, 3 op/s 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]': finished 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.35360 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1"}]': finished 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1007690566' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-74307-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/230398496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: osdmap e90: 8 total, 8 up, 8 in 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.25411 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-74307-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.25429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:58.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-74719-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]': finished 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]': finished 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.25411 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-74307-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.25429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: osdmap e91: 8 total, 8 up, 8 in 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "overlaypool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "overlaypool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3802736140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:58.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:57 smithi181 ceph-mon[30237]: from='client.25606 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: pgmap v91: 796 pgs: 280 unknown, 516 active+clean; 145 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 5.2 KiB/s wr, 3 op/s 2024-04-27T01:12:58.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]': finished 2024-04-27T01:12:58.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.35360 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1"}]': finished 2024-04-27T01:12:58.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:58.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:12:58.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1007690566' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-74307-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/230398496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: osdmap e90: 8 total, 8 up, 8 in 2024-04-27T01:12:58.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:58.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:12:58.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.25411 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-74307-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.25429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:58.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-74719-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]': finished 2024-04-27T01:12:58.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]': finished 2024-04-27T01:12:58.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:12:58.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.25411 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-74307-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:58.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.25429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: osdmap e91: 8 total, 8 up, 8 in 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "overlaypool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "overlaypool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3802736140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:58.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[37680]: from='client.25606 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: pgmap v91: 796 pgs: 280 unknown, 516 active+clean; 145 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 5.2 KiB/s wr, 3 op/s 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi162-74415-10"}]': finished 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.35360 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi162-74536-1","app":"app1","key":"key1"}]': finished 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2959573279' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1007690566' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-74307-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/230398496' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: osdmap e90: 8 total, 8 up, 8 in 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.35267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]: dispatch 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:12:58.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.25411 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-74307-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.25429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi162-74719-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]': finished 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.35267 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi162-74415-10"}]': finished 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.25411 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi162-74307-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.25429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: osdmap e91: 8 total, 8 up, 8 in 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "overlaypool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "overlaypool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:58.093 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:58.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:58.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3802736140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:58.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:58.094 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:57 smithi162 ceph-mon[26714]: from='client.25606 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:12:58.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: Running main() from gmock_main.cc 2024-04-27T01:12:58.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-04-27T01:12:58.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] Global test environment set-up. 2024-04-27T01:12:58.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-04-27T01:12:58.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-04-27T01:12:58.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStat.Stat (309 ms) 2024-04-27T01:12:58.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-04-27T01:12:58.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (22 ms) 2024-04-27T01:12:58.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-04-27T01:12:58.857 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStat.StatNS (69 ms) 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (3 ms) 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] 5 tests from LibRadosStat (403 ms total) 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (2226 ms) 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (18 ms) 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-04-27T01:12:58.858 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (2 ms) 2024-04-27T01:12:58.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-04-27T01:12:58.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (8 ms) 2024-04-27T01:12:58.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (2254 ms total) 2024-04-27T01:12:58.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: 2024-04-27T01:12:58.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [----------] Global test environment tear-down 2024-04-27T01:12:58.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (26022 ms total) 2024-04-27T01:12:58.859 INFO:tasks.workunit.client.0.smithi162.stdout: api_stat: [ PASSED ] 9 tests. 2024-04-27T01:12:58.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-04-27T01:12:58.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-04-27T01:12:58.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-04-27T01:12:58.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-04-27T01:12:58.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: seed 74448 2024-04-27T01:12:58.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-04-27T01:12:58.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (312 ms) 2024-04-27T01:12:58.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-04-27T01:12:58.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (4 ms) 2024-04-27T01:12:58.867 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1028 ms) 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1016 ms) 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5 ms) 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (15 ms) 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (15 ms) 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2399 ms total) 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-04-27T01:12:58.868 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (861 ms) 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (2 ms) 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1004 ms) 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1093 ms) 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (28 ms) 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (14 ms) 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (12 ms) 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-04-27T01:12:58.869 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (12 ms) 2024-04-27T01:12:58.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3026 ms total) 2024-04-27T01:12:58.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: 2024-04-27T01:12:58.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-04-27T01:12:58.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (26150 ms total) 2024-04-27T01:12:58.870 INFO:tasks.workunit.client.0.smithi162.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-04-27T01:12:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: pgmap v94: 624 pgs: 72 creating+peering, 552 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.2 KiB/s wr, 3 op/s 2024-04-27T01:12:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "overlaypool": "test-rados-api-smithi162-74739-6"}]': finished 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi162-75564-2"}]': finished 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]': finished 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]': finished 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.25606 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: osdmap e92: 8 total, 8 up, 8 in 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74739-6", "mode": "writeback"}]: dispatch 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:59.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74739-6", "mode": "writeback"}]: dispatch 2024-04-27T01:12:59.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:12:58 smithi181 ceph-mon[30237]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:59.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: pgmap v94: 624 pgs: 72 creating+peering, 552 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.2 KiB/s wr, 3 op/s 2024-04-27T01:12:59.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:59.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "overlaypool": "test-rados-api-smithi162-74739-6"}]': finished 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi162-75564-2"}]': finished 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]': finished 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]': finished 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.25606 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: osdmap e92: 8 total, 8 up, 8 in 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74739-6", "mode": "writeback"}]: dispatch 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:59.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74739-6", "mode": "writeback"}]: dispatch 2024-04-27T01:12:59.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[26714]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:59.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: pgmap v94: 624 pgs: 72 creating+peering, 552 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 6.2 KiB/s wr, 3 op/s 2024-04-27T01:12:59.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:12:59.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:12:59.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "overlaypool": "test-rados-api-smithi162-74739-6"}]': finished 2024-04-27T01:12:59.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi162-75564-2"}]': finished 2024-04-27T01:12:59.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi162-74448-10"}]': finished 2024-04-27T01:12:59.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi162-74772-7"}]': finished 2024-04-27T01:12:59.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.25606 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:12:59.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: osdmap e92: 8 total, 8 up, 8 in 2024-04-27T01:12:59.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1174468446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:12:59.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2661600190' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:59.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2559629892' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:59.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74739-6", "mode": "writeback"}]: dispatch 2024-04-27T01:12:59.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.35342 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]: dispatch 2024-04-27T01:12:59.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.35405 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi162-75564-2"}]: dispatch 2024-04-27T01:12:59.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74739-6", "mode": "writeback"}]: dispatch 2024-04-27T01:12:59.091 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:12:58 smithi162 ceph-mon[37680]: from='client.25297 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]: dispatch 2024-04-27T01:13:00.038 INFO:tasks.workunit.client.0.smithi162.stdout: pool: Running main() from gmock_main.cc 2024-04-27T01:13:00.038 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [----------] Global test environment set-up. 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ OK ] NeoRadosPools.PoolList (8566 ms) 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4845 ms) 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (4212 ms) 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4889 ms) 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2131 ms) 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-04-27T01:13:00.039 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2294 ms) 2024-04-27T01:13:00.040 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [----------] 6 tests from NeoRadosPools (26937 ms total) 2024-04-27T01:13:00.040 INFO:tasks.workunit.client.0.smithi162.stdout: pool: 2024-04-27T01:13:00.040 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [----------] Global test environment tear-down 2024-04-27T01:13:00.040 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [==========] 6 tests from 1 test suite ran. (26937 ms total) 2024-04-27T01:13:00.040 INFO:tasks.workunit.client.0.smithi162.stdout: pool: [ PASSED ] 6 tests. 2024-04-27T01:13:00.091 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:12:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:12:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: pool 'PoolQuotaPP_smithi162-74325-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]': finished 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi162-75564-2"}]': finished 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74739-6", "mode": "writeback"}]': finished 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]': finished 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: osdmap e93: 8 total, 8 up, 8 in 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1897087921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.27415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:00.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:00.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4246459859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-74726-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:00.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:00.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.26254 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-74726-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:00.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi162-75564-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:00.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:00.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:00.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:00.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: pool 'PoolQuotaPP_smithi162-74325-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-27T01:13:00.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]': finished 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi162-75564-2"}]': finished 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74739-6", "mode": "writeback"}]': finished 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]': finished 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: osdmap e93: 8 total, 8 up, 8 in 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1897087921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.27415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4246459859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-74726-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.26254 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-74726-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi162-75564-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:00.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:01.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:01.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: pool 'PoolQuotaPP_smithi162-74325-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.35342 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi162-74448-10"}]': finished 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.35405 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi162-75564-2"}]': finished 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74739-6", "mode": "writeback"}]': finished 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.25297 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi162-74772-7"}]': finished 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: osdmap e93: 8 total, 8 up, 8 in 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1897087921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.27415 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4246459859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-74726-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.26254 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-74726-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:01.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi162-75564-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:01.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:01.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify_test_cb 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (375 ms) 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94282394001568 err -107 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (10 ms) 2024-04-27T01:13:01.173 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94282394017472 err -107 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (14 ms) 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24715 notify_id 330712481793 cookie 94282394021904 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (11 ms) 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24715 notify_id 330712481792 cookie 94282394211152 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (8 ms) 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24715 notify_id 330712481792 cookie 94282394185376 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (9 ms) 2024-04-27T01:13:01.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24715 notify_id 330712481793 cookie 94282394185376 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24715 notify_id 330712481793 cookie 94282394227488 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (10 ms) 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24715 notify_id 330712481793 cookie 94282394185376 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24715 notify_id 335007449090 cookie 94282394185376 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3008 ms) 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94282394227488 err -107 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_cb from 24715 notify_id 360777252867 cookie 94282394227488 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-04-27T01:13:01.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94282394227488 err -107 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1004 ms) 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9456 ms total) 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: watch_notify_test_cb 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1167 ms) 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1167 ms total) 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-04-27T01:13:01.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (28359 ms total) 2024-04-27T01:13:01.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-04-27T01:13:02.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-04-27T01:13:02.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-04-27T01:13:02.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-04-27T01:13:02.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-04-27T01:13:02.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-04-27T01:13:02.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify 2024-04-27T01:13:02.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1160 ms) 2024-04-27T01:13:02.174 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (1140 ms) 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (2300 ms total) 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (88 ms) 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (5289 ms) 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-04-27T01:13:02.175 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (45 ms) 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (59 ms) 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94801259127136 notify_id 399431958531 notifier_gid 35408 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (61 ms) 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94801259127136 notify_id 399431958532 notifier_gid 35408 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (8 ms) 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94801259127136 notify_id 399431958532 notifier_gid 35408 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (29 ms) 2024-04-27T01:13:02.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94801259149776 notify_id 399431958528 notifier_gid 35408 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (7 ms) 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94801259149696 notify_id 399431958529 notifier_gid 35408 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (22 ms) 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94801259149696 notify_id 399431958530 notifier_gid 35408 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (12 ms) 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: trying... 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94801259149696 notify_id 399431958533 notifier_gid 35408 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: timed out 2024-04-27T01:13:02.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: flushing 2024-04-27T01:13:02.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:13:02.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:13:02.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4210 ms) 2024-04-27T01:13:02.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ RUN ] LibRadosService.Status 2024-04-27T01:13:02.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ OK ] LibRadosService.Status (20043 ms) 2024-04-27T01:13:02.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [----------] 4 tests from LibRadosService (29308 ms total) 2024-04-27T01:13:02.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: 2024-04-27T01:13:02.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [----------] Global test environment tear-down 2024-04-27T01:13:02.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29308 ms total) 2024-04-27T01:13:02.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_service: [ PASSED ] 4 tests. 2024-04-27T01:13:02.493 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:13:02.493 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: pgmap v97: 584 pgs: 192 unknown, 392 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 4.5 KiB/s wr, 0 op/s 2024-04-27T01:13:02.493 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi162-75564-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:02.493 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.27415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:02.493 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]': finished 2024-04-27T01:13:02.494 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.26254 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-74726-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:02.494 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:02.494 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/903525055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-74307-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.494 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: osdmap e94: 8 total, 8 up, 8 in 2024-04-27T01:13:02.494 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:02.494 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.38288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-74307-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.494 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:02.494 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi162-75564-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:02.494 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.38288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-74307-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:02.494 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]': finished 2024-04-27T01:13:02.495 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: osdmap e95: 8 total, 8 up, 8 in 2024-04-27T01:13:02.495 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi162-75564-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:02.495 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.495 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1078270362' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.495 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi162-75564-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:02.495 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.495 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.28540 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.495 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:02.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: pgmap v97: 584 pgs: 192 unknown, 392 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 4.5 KiB/s wr, 0 op/s 2024-04-27T01:13:02.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi162-75564-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:02.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.27415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:02.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]': finished 2024-04-27T01:13:02.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.26254 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-74726-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:02.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:02.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/903525055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-74307-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: osdmap e94: 8 total, 8 up, 8 in 2024-04-27T01:13:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.38288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-74307-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:02.592 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi162-75564-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.38288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-74307-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]': finished 2024-04-27T01:13:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: osdmap e95: 8 total, 8 up, 8 in 2024-04-27T01:13:02.593 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi162-75564-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:02.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1078270362' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi162-75564-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:02.594 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.28540 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.595 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:02.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: pgmap v97: 584 pgs: 192 unknown, 392 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 4.5 KiB/s wr, 0 op/s 2024-04-27T01:13:02.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi162-75564-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:02.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.27415 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:02.596 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi162-74719-12"}]': finished 2024-04-27T01:13:02.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.26254 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi162-74726-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:02.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1186571600' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:02.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/903525055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-74307-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: osdmap e94: 8 total, 8 up, 8 in 2024-04-27T01:13:02.597 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:02.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.38288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-74307-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.25378 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]: dispatch 2024-04-27T01:13:02.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi162-75564-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:02.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.38288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi162-74307-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:02.598 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.25378 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi162-74719-12"}]': finished 2024-04-27T01:13:02.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: osdmap e95: 8 total, 8 up, 8 in 2024-04-27T01:13:02.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi162-75564-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:02.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1078270362' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi162-75564-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:02.599 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.600 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.28540 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:02.600 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:03.187 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [----------] Global test environment set-up. 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5046 ms) 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (50 ms) 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20032 ms) 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: attempt 0 of 20 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5149 ms) 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30277 ms total) 2024-04-27T01:13:03.188 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: 2024-04-27T01:13:03.189 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [----------] Global test environment tear-down 2024-04-27T01:13:03.189 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30277 ms total) 2024-04-27T01:13:03.189 INFO:tasks.workunit.client.0.smithi162.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-04-27T01:13:03.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[26714]: pgmap v99: 616 pgs: 224 unknown, 392 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-27T01:13:03.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:03.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[26714]: from='client.28540 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:03.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:03.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[26714]: osdmap e96: 8 total, 8 up, 8 in 2024-04-27T01:13:03.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/544821641' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-74682-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:03.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[37680]: pgmap v99: 616 pgs: 224 unknown, 392 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-27T01:13:03.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:03.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[37680]: from='client.28540 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:03.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:03.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[37680]: osdmap e96: 8 total, 8 up, 8 in 2024-04-27T01:13:03.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/544821641' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-74682-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:03 smithi181 ceph-mon[30237]: pgmap v99: 616 pgs: 224 unknown, 392 active+clean; 145 MiB data, 775 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-27T01:13:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:03 smithi181 ceph-mon[30237]: from='client.28540 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:03 smithi181 ceph-mon[30237]: osdmap e96: 8 total, 8 up, 8 in 2024-04-27T01:13:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/544821641' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-74682-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:04.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: pgmap v102: 616 pgs: 96 creating+peering, 96 unknown, 424 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 954 B/s rd, 3.0 KiB/s wr, 3 op/s 2024-04-27T01:13:04.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4292481105' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-27T01:13:04.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-27T01:13:04.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi162-75564-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi162-75564-3"}]': finished 2024-04-27T01:13:04.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: osdmap e97: 8 total, 8 up, 8 in 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1094314586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='client.40466 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='client.30847 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-74682-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:04.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: pgmap v102: 616 pgs: 96 creating+peering, 96 unknown, 424 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 954 B/s rd, 3.0 KiB/s wr, 3 op/s 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4292481105' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi162-75564-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi162-75564-3"}]': finished 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: osdmap e97: 8 total, 8 up, 8 in 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1094314586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='client.40466 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='client.30847 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-74682-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:04.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:04.593 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:04 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: pgmap v102: 616 pgs: 96 creating+peering, 96 unknown, 424 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 954 B/s rd, 3.0 KiB/s wr, 3 op/s 2024-04-27T01:13:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4292481105' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.162:0/509891751' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi162-75564-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi162-75564-3"}]': finished 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: osdmap e97: 8 total, 8 up, 8 in 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1094314586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='client.40466 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='client.30847 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-74682-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:04 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:05 smithi181 ceph-mon[30237]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:13:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:05 smithi181 ceph-mon[30237]: from='client.40466 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:05 smithi181 ceph-mon[30237]: from='client.30847 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-74682-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:05 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:05 smithi181 ceph-mon[30237]: osdmap e98: 8 total, 8 up, 8 in 2024-04-27T01:13:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1194189807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-74307-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:05 smithi181 ceph-mon[30237]: from='client.41957 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-74307-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:06.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:06.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:06.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[37680]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:13:06.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[37680]: from='client.40466 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:06.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[37680]: from='client.30847 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-74682-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:06.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:06.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[37680]: osdmap e98: 8 total, 8 up, 8 in 2024-04-27T01:13:06.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1194189807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-74307-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[37680]: from='client.41957 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-74307-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[26714]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[26714]: from='client.40466 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[26714]: from='client.30847 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi162-74682-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[26714]: osdmap e98: 8 total, 8 up, 8 in 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1194189807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-74307-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[26714]: from='client.41957 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-74307-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:06.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:06.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:07.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:06 smithi181 ceph-mon[30237]: pgmap v105: 624 pgs: 32 creating+peering, 200 unknown, 392 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2 op/s 2024-04-27T01:13:07.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:07.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:06 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:07.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:06 smithi181 ceph-mon[30237]: from='client.41957 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-74307-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:07.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:06 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:07.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:06 smithi181 ceph-mon[30237]: osdmap e99: 8 total, 8 up, 8 in 2024-04-27T01:13:07.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/289014277' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:07.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:06 smithi181 ceph-mon[30237]: from='client.34438 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:07.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:06 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:07.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[26714]: pgmap v105: 624 pgs: 32 creating+peering, 200 unknown, 392 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2 op/s 2024-04-27T01:13:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[26714]: from='client.41957 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-74307-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[26714]: osdmap e99: 8 total, 8 up, 8 in 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/289014277' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[26714]: from='client.34438 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[37680]: pgmap v105: 624 pgs: 32 creating+peering, 200 unknown, 392 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2 op/s 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[37680]: from='client.41957 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi162-74307-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[37680]: osdmap e99: 8 total, 8 up, 8 in 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/289014277' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:07.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[37680]: from='client.34438 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:07.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:07.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:08.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:08.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[37680]: from='client.34438 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:08.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:08.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2072660249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:08.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[37680]: osdmap e100: 8 total, 8 up, 8 in 2024-04-27T01:13:08.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:08.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:08.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[37680]: from='client.35953 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:08.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:08.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[26714]: from='client.34438 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:08.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:08.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2072660249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:08.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[26714]: osdmap e100: 8 total, 8 up, 8 in 2024-04-27T01:13:08.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:08.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:08.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:08 smithi162 ceph-mon[26714]: from='client.35953 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:08 smithi181 ceph-mon[30237]: from='client.34438 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2072660249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:08 smithi181 ceph-mon[30237]: osdmap e100: 8 total, 8 up, 8 in 2024-04-27T01:13:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:08 smithi181 ceph-mon[30237]: from='client.35953 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:09.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: pgmap v108: 624 pgs: 1 active, 56 creating+activating, 68 creating+peering, 56 unknown, 443 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-27T01:13:09.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-7"}]': finished 2024-04-27T01:13:09.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: from='client.35953 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:09.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: osdmap e101: 8 total, 8 up, 8 in 2024-04-27T01:13:09.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2199239894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-74307-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:09.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: from='client.45518 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-74307-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:09.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:09.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:09.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:13:09.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:09.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[37680]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:09.849 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: pgmap v108: 624 pgs: 1 active, 56 creating+activating, 68 creating+peering, 56 unknown, 443 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-27T01:13:09.849 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-7"}]': finished 2024-04-27T01:13:09.849 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: from='client.35953 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:09.849 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: osdmap e101: 8 total, 8 up, 8 in 2024-04-27T01:13:09.849 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2199239894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-74307-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:09.849 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: from='client.45518 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-74307-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:09.850 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:09.850 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:09.850 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:13:09.850 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:09.850 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-mon[26714]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:09.850 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:13:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:13:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:13:10.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: pgmap v108: 624 pgs: 1 active, 56 creating+activating, 68 creating+peering, 56 unknown, 443 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-27T01:13:10.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-7"}]': finished 2024-04-27T01:13:10.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: from='client.35953 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:10.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: osdmap e101: 8 total, 8 up, 8 in 2024-04-27T01:13:10.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2199239894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-74307-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:10.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: from='client.45518 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-74307-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:10.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:10.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:13:10.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:13:10.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:10.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:09 smithi181 ceph-mon[30237]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:11.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: pgmap v110: 592 pgs: 51 creating+peering, 109 unknown, 432 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-27T01:13:11.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:11.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.45518 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-74307-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/969962502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2465159159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: osdmap e102: 8 total, 8 up, 8 in 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.37714 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.37735 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.37714 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi162-75564-3"}]': finished 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.37735 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:11.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:11.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: osdmap e103: 8 total, 8 up, 8 in 2024-04-27T01:13:11.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:11.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/263931825' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.38869 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:11.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: pgmap v110: 592 pgs: 51 creating+peering, 109 unknown, 432 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.45518 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-74307-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/969962502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2465159159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: osdmap e102: 8 total, 8 up, 8 in 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.37714 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.37735 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:11.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.37714 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi162-75564-3"}]': finished 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.37735 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: osdmap e103: 8 total, 8 up, 8 in 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/263931825' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.38869 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:11.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: pgmap v110: 592 pgs: 51 creating+peering, 109 unknown, 432 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-27T01:13:11.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:11.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.45518 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi162-74307-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:11.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/969962502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2465159159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: osdmap e102: 8 total, 8 up, 8 in 2024-04-27T01:13:11.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.37714 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.37735 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.37714 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-75013-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi162-75564-3"}]': finished 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.37735 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/96264662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: osdmap e103: 8 total, 8 up, 8 in 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7"}]: dispatch 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.28486 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi162-75564-3"}]: dispatch 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/263931825' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.38869 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:11.342 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:11.414 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: Running main() from gmock_main.cc 2024-04-27T01:13:11.414 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-04-27T01:13:11.414 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [----------] Global test environment set-up. 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (18637 ms) 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (7155 ms) 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (12545 ms) 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [----------] 3 tests from NeoradosECList (38337 ms total) 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [----------] Global test environment tear-down 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (38338 ms total) 2024-04-27T01:13:11.415 INFO:tasks.workunit.client.0.smithi162.stdout: ec_list: [ PASSED ] 3 tests. 2024-04-27T01:13:12.260 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:13:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:13:12.275 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4"}]: dispatch 2024-04-27T01:13:12.275 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:11 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4"}]: dispatch 2024-04-27T01:13:12.275 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:11 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7"}]': finished 2024-04-27T01:13:12.275 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:11 smithi181 ceph-mon[30237]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi162-75564-3"}]': finished 2024-04-27T01:13:12.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:11 smithi181 ceph-mon[30237]: from='client.38869 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:12.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:11 smithi181 ceph-mon[30237]: osdmap e104: 8 total, 8 up, 8 in 2024-04-27T01:13:12.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2305620290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-74307-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:12.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:11 smithi181 ceph-mon[30237]: from='client.39970 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-74307-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:12.276 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:12.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4"}]: dispatch 2024-04-27T01:13:12.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4"}]: dispatch 2024-04-27T01:13:12.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7"}]': finished 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[26714]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi162-75564-3"}]': finished 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[26714]: from='client.38869 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[26714]: osdmap e104: 8 total, 8 up, 8 in 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2305620290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-74307-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[26714]: from='client.39970 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-74307-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4"}]: dispatch 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4"}]: dispatch 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-7"}]': finished 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[37680]: from='client.28486 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi162-75564-3"}]': finished 2024-04-27T01:13:12.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[37680]: from='client.38869 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi162-74325-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:12.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[37680]: osdmap e104: 8 total, 8 up, 8 in 2024-04-27T01:13:12.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2305620290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-74307-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:12.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[37680]: from='client.39970 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-74307-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:12.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:13.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:13 smithi181 ceph-mon[30237]: pgmap v113: 648 pgs: 25 creating+peering, 199 unknown, 424 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:13:13.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:13 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4"}]': finished 2024-04-27T01:13:13.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:13 smithi181 ceph-mon[30237]: from='client.39970 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-74307-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:13.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:13 smithi181 ceph-mon[30237]: osdmap e105: 8 total, 8 up, 8 in 2024-04-27T01:13:13.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:13:13.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:13.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:13 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:13:13.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[26714]: pgmap v113: 648 pgs: 25 creating+peering, 199 unknown, 424 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:13:13.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4"}]': finished 2024-04-27T01:13:13.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[26714]: from='client.39970 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-74307-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:13.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[26714]: osdmap e105: 8 total, 8 up, 8 in 2024-04-27T01:13:13.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:13:13.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:13.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:13:13.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[37680]: pgmap v113: 648 pgs: 25 creating+peering, 199 unknown, 424 active+clean; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:13:13.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4"}]': finished 2024-04-27T01:13:13.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[37680]: from='client.39970 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi162-74307-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:13.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[37680]: osdmap e105: 8 total, 8 up, 8 in 2024-04-27T01:13:13.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:13:13.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:13.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:13 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6"}]: dispatch 2024-04-27T01:13:15.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:15.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:13:15.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6"}]': finished 2024-04-27T01:13:15.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi162-74739-6", "pool2": "test-rados-api-smithi162-74739-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:15.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: osdmap e106: 8 total, 8 up, 8 in 2024-04-27T01:13:15.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:15.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi162-74739-6", "pool2": "test-rados-api-smithi162-74739-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: osdmap e107: 8 total, 8 up, 8 in 2024-04-27T01:13:15.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3069030620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-74325-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:14 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:15.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:13:15.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6"}]': finished 2024-04-27T01:13:15.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi162-74739-6", "pool2": "test-rados-api-smithi162-74739-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:15.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: osdmap e106: 8 total, 8 up, 8 in 2024-04-27T01:13:15.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:15.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi162-74739-6", "pool2": "test-rados-api-smithi162-74739-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: osdmap e107: 8 total, 8 up, 8 in 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3069030620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-74325-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi162-74739-4", "tierpool": "test-rados-api-smithi162-74739-6"}]': finished 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1411275433' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi162-74739-6", "pool2": "test-rados-api-smithi162-74739-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: osdmap e106: 8 total, 8 up, 8 in 2024-04-27T01:13:15.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:15.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi162-74739-6", "pool2": "test-rados-api-smithi162-74739-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: osdmap e107: 8 total, 8 up, 8 in 2024-04-27T01:13:15.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3069030620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-74325-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:15.341 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:14 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:16.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: pgmap v116: 616 pgs: 64 creating+peering, 32 unknown, 520 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-27T01:13:16.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:16.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:13:16.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:13:16.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:13:16.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:13:16.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.42223 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-74325-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:16.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:13:16.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:13:16.262 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi162-74739-6", "pool2": "test-rados-api-smithi162-74739-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.42223 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-74325-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1023676359' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/437875327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-74307-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:16.263 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:15 smithi181 ceph-mon[30237]: osdmap e108: 8 total, 8 up, 8 in 2024-04-27T01:13:16.284 INFO:tasks.workunit.client.0.smithi162.stdout:api_watch_notify_pp: flushed 2024-04-27T01:13:16.284 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3014 ms) 2024-04-27T01:13:16.284 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-04-27T01:13:16.284 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: trying... 2024-04-27T01:13:16.284 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94801259149696 notify_id 408021893125 notifier_gid 35408 2024-04-27T01:13:16.284 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: timed out 2024-04-27T01:13:16.284 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: flushing 2024-04-27T01:13:16.284 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: flushed 2024-04-27T01:13:16.284 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3016 ms) 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: List watches 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify2 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94801259149696 notify_id 420906795014 notifier_gid 35408 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify2 done 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: watch_check 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: unwatch2 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: flushing 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: done 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3031 ms) 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: List watches 2024-04-27T01:13:16.285 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify2 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: handle_notify cookie 94801259149696 notify_id 433791696902 notifier_gid 35408 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: notify2 done 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: watch_check 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: unwatch2 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: flushing 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: done 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3016 ms) 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (17700 ms total) 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (43471 ms total) 2024-04-27T01:13:16.286 INFO:tasks.workunit.client.0.smithi162.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-04-27T01:13:16.287 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: Running main() from gmock_main.cc 2024-04-27T01:13:16.287 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-04-27T01:13:16.287 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [----------] Global test environment set-up. 2024-04-27T01:13:16.287 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-04-27T01:13:16.287 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-04-27T01:13:16.287 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (13096 ms) 2024-04-27T01:13:16.287 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-04-27T01:13:16.287 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (4514 ms) 2024-04-27T01:13:16.287 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3103 ms) 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (5024 ms) 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3905 ms) 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (4257 ms) 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (9280 ms) 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (43180 ms total) 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [----------] Global test environment tear-down 2024-04-27T01:13:16.288 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (43180 ms total) 2024-04-27T01:13:16.289 INFO:tasks.workunit.client.0.smithi162.stdout: write_operations: [ PASSED ] 7 tests. 2024-04-27T01:13:16.305 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (11225 ms) 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (7890 ms) 2024-04-27T01:13:16.306 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3953 ms) 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2919 ms) 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (5595 ms) 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3646 ms) 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (8191 ms) 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (43419 ms total) 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (43419 ms total) 2024-04-27T01:13:16.307 INFO:tasks.workunit.client.0.smithi162.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-04-27T01:13:16.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: pgmap v116: 616 pgs: 64 creating+peering, 32 unknown, 520 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.42223 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-74325-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:13:16.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi162-74739-6", "pool2": "test-rados-api-smithi162-74739-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T01:13:16.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.42223 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-74325-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1023676359' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/437875327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-74307-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:16.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[37680]: osdmap e108: 8 total, 8 up, 8 in 2024-04-27T01:13:16.346 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: pgmap v116: 616 pgs: 64 creating+peering, 32 unknown, 520 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-27T01:13:16.346 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:16.346 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.0"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.1"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.42223 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-74325-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.2"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.3"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.4"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:13:16.347 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.5"}]: dispatch 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.6"}]: dispatch 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.7"}]: dispatch 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.8"}]: dispatch 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "20.9"}]: dispatch 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.35408 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi162-74739-6", "pool2": "test-rados-api-smithi162-74739-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.42223 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi162-74325-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1023676359' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:16.348 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/437875327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-74307-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:16.349 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:16.349 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:16.349 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:15 smithi162 ceph-mon[26714]: osdmap e108: 8 total, 8 up, 8 in 2024-04-27T01:13:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: pgmap v119: 584 pgs: 128 unknown, 456 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-04-27T01:13:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: 20.0 deep-scrub starts 2024-04-27T01:13:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: 20.0 deep-scrub ok 2024-04-27T01:13:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: 20.1 deep-scrub starts 2024-04-27T01:13:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: 20.1 deep-scrub ok 2024-04-27T01:13:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: 20.2 deep-scrub starts 2024-04-27T01:13:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: 20.2 deep-scrub ok 2024-04-27T01:13:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: from='client.43330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: 20.6 deep-scrub starts 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: 20.6 deep-scrub ok 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: 20.3 deep-scrub starts 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: 20.3 deep-scrub ok 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: from='client.43810 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-74307-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: from='client.43330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: from='client.43810 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-74307-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: osdmap e109: 8 total, 8 up, 8 in 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:17.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:17.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:17.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: pgmap v119: 584 pgs: 128 unknown, 456 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: 20.0 deep-scrub starts 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: 20.0 deep-scrub ok 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: 20.1 deep-scrub starts 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: 20.1 deep-scrub ok 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: 20.2 deep-scrub starts 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: 20.2 deep-scrub ok 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: from='client.43330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: 20.6 deep-scrub starts 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: 20.6 deep-scrub ok 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: 20.3 deep-scrub starts 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: 20.3 deep-scrub ok 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: from='client.43810 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-74307-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: from='client.43330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:17.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: from='client.43810 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-74307-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: osdmap e109: 8 total, 8 up, 8 in 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: pgmap v119: 584 pgs: 128 unknown, 456 active+clean; 145 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: 20.0 deep-scrub starts 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: 20.0 deep-scrub ok 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: 20.1 deep-scrub starts 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: 20.1 deep-scrub ok 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: 20.2 deep-scrub starts 2024-04-27T01:13:17.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: 20.2 deep-scrub ok 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: from='client.43330 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: 20.6 deep-scrub starts 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: 20.6 deep-scrub ok 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: 20.3 deep-scrub starts 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: 20.3 deep-scrub ok 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: from='client.43810 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-74307-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: from='client.43330 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: from='client.43810 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi162-74307-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: osdmap e109: 8 total, 8 up, 8 in 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:17.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:17.842 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: 20.8 deep-scrub starts 2024-04-27T01:13:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: 20.8 deep-scrub ok 2024-04-27T01:13:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: 20.7 deep-scrub starts 2024-04-27T01:13:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: 20.7 deep-scrub ok 2024-04-27T01:13:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: 20.5 deep-scrub starts 2024-04-27T01:13:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: 20.5 deep-scrub ok 2024-04-27T01:13:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: pgmap v122: 548 pgs: 64 creating+peering, 484 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:13:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: 20.9 deep-scrub starts 2024-04-27T01:13:18.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-9"}]': finished 2024-04-27T01:13:18.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-9", "mode": "writeback"}]: dispatch 2024-04-27T01:13:18.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: osdmap e110: 8 total, 8 up, 8 in 2024-04-27T01:13:18.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-9", "mode": "writeback"}]: dispatch 2024-04-27T01:13:18.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1063773004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-74325-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:18.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: from='client.46045 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-74325-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:18.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: 20.9 deep-scrub ok 2024-04-27T01:13:18.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:18.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: 20.4 deep-scrub starts 2024-04-27T01:13:18.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:18 smithi181 ceph-mon[30237]: 20.4 deep-scrub ok 2024-04-27T01:13:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: 20.8 deep-scrub starts 2024-04-27T01:13:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: 20.8 deep-scrub ok 2024-04-27T01:13:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: 20.7 deep-scrub starts 2024-04-27T01:13:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: 20.7 deep-scrub ok 2024-04-27T01:13:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: 20.5 deep-scrub starts 2024-04-27T01:13:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: 20.5 deep-scrub ok 2024-04-27T01:13:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: pgmap v122: 548 pgs: 64 creating+peering, 484 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: 20.9 deep-scrub starts 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-9"}]': finished 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-9", "mode": "writeback"}]: dispatch 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: osdmap e110: 8 total, 8 up, 8 in 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-9", "mode": "writeback"}]: dispatch 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1063773004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-74325-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: from='client.46045 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-74325-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: 20.9 deep-scrub ok 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: 20.4 deep-scrub starts 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[37680]: 20.4 deep-scrub ok 2024-04-27T01:13:18.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: 20.8 deep-scrub starts 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: 20.8 deep-scrub ok 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: 20.7 deep-scrub starts 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: 20.7 deep-scrub ok 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: 20.5 deep-scrub starts 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: 20.5 deep-scrub ok 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: pgmap v122: 548 pgs: 64 creating+peering, 484 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: 20.9 deep-scrub starts 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-9"}]': finished 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-9", "mode": "writeback"}]: dispatch 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: osdmap e110: 8 total, 8 up, 8 in 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-9", "mode": "writeback"}]: dispatch 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1063773004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-74325-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: from='client.46045 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-74325-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: 20.9 deep-scrub ok 2024-04-27T01:13:18.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:18.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: 20.4 deep-scrub starts 2024-04-27T01:13:18.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:18 smithi162 ceph-mon[26714]: 20.4 deep-scrub ok 2024-04-27T01:13:20.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:20.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-9", "mode": "writeback"}]': finished 2024-04-27T01:13:20.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.46045 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-74325-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:20.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: osdmap e111: 8 total, 8 up, 8 in 2024-04-27T01:13:20.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: osdmap e112: 8 total, 8 up, 8 in 2024-04-27T01:13:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2679281963' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-74307-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.56234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-74307-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[37680]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-9", "mode": "writeback"}]': finished 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.46045 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-74325-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: osdmap e111: 8 total, 8 up, 8 in 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: osdmap e112: 8 total, 8 up, 8 in 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2679281963' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-74307-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:20.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.56234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-74307-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:20.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:20.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-mon[26714]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.091 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:13:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:13:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:13:20.092 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] Global test environment set-up. 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-04-27T01:12:32.831+0000 7f4b63036780 -1 failed for service _ceph-mon._tcp 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-04-27T01:12:32.831+0000 7f4b63036780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (72 ms) 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2013 ms) 2024-04-27T01:13:20.093 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2085 ms total) 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: started 0x5594d2713660 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: started 0x5594d2ad8ae0 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: started 2 aios 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: waiting 0x5594d2713660 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: waiting 0x5594d2ad8ae0 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: done. 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (13124 ms) 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (13124 ms total) 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-04-27T01:13:20.094 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.Exec (119 ms) 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (5 ms) 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (5 ms) 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4756 ms) 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-04-27T01:13:20.095 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (23096 ms) 2024-04-27T01:13:20.096 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] 8 tests from LibRadosMisc (27981 ms total) 2024-04-27T01:13:20.096 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: 2024-04-27T01:13:20.096 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [----------] Global test environment tear-down 2024-04-27T01:13:20.096 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (47324 ms total) 2024-04-27T01:13:20.096 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc: [ PASSED ] 12 tests. 2024-04-27T01:13:20.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:20.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-9", "mode": "writeback"}]': finished 2024-04-27T01:13:20.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.46045 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi162-74325-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:20.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: osdmap e111: 8 total, 8 up, 8 in 2024-04-27T01:13:20.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: osdmap e112: 8 total, 8 up, 8 in 2024-04-27T01:13:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2679281963' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-74307-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.56234 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-74307-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:20.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:19 smithi181 ceph-mon[30237]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:21.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: pgmap v126: 484 pgs: 128 unknown, 356 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail; 19 MiB/s wr, 0 op/s 2024-04-27T01:13:21.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:21.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.56234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-74307-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-74726-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: osdmap e113: 8 total, 8 up, 8 in 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-74726-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: pgmap v126: 484 pgs: 128 unknown, 356 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail; 19 MiB/s wr, 0 op/s 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.56234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-74307-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:21.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-74726-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:21.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: osdmap e113: 8 total, 8 up, 8 in 2024-04-27T01:13:21.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:21.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:21.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-74726-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:21.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:21.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:21.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: pgmap v126: 484 pgs: 128 unknown, 356 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail; 19 MiB/s wr, 0 op/s 2024-04-27T01:13:21.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:21.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:21.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:21.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:21.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.56234 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi162-74307-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-74726-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: osdmap e113: 8 total, 8 up, 8 in 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9"}]: dispatch 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-74726-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:21.261 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:22.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:13:22.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9"}]': finished 2024-04-27T01:13:22.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[26714]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:22.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[26714]: osdmap e114: 8 total, 8 up, 8 in 2024-04-27T01:13:22.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1435625223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-74325-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:22.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[26714]: from='client.56363 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-74325-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:22.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:13:22.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9"}]': finished 2024-04-27T01:13:22.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[37680]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:22.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[37680]: osdmap e114: 8 total, 8 up, 8 in 2024-04-27T01:13:22.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1435625223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-74325-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:22.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:22 smithi162 ceph-mon[37680]: from='client.56363 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-74325-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:22.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:13:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:13:22.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:22 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:13:22.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:22 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-9"}]': finished 2024-04-27T01:13:22.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:22 smithi181 ceph-mon[30237]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74627-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:22.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:22 smithi181 ceph-mon[30237]: osdmap e114: 8 total, 8 up, 8 in 2024-04-27T01:13:22.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1435625223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-74325-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:22.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:22 smithi181 ceph-mon[30237]: from='client.56363 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-74325-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:23.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: pgmap v129: 484 pgs: 160 unknown, 324 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:13:23.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:23.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-74726-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]': finished 2024-04-27T01:13:23.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: from='client.56363 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-74325-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:23.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: osdmap e115: 8 total, 8 up, 8 in 2024-04-27T01:13:23.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache", "force_nonempty":""}]: dispatch 2024-04-27T01:13:23.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3213881221' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-74307-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:23.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache", "force_nonempty":""}]: dispatch 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: from='client.56378 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-74307-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: pgmap v129: 484 pgs: 160 unknown, 324 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-74726-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]': finished 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: from='client.56363 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-74325-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: osdmap e115: 8 total, 8 up, 8 in 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache", "force_nonempty":""}]: dispatch 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3213881221' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-74307-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:23.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache", "force_nonempty":""}]: dispatch 2024-04-27T01:13:23.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: from='client.56378 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-74307-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:23.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:23.340 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:23.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: pgmap v129: 484 pgs: 160 unknown, 324 active+clean; 217 MiB data, 1024 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:13:23.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:23.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi162-74726-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]': finished 2024-04-27T01:13:23.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: from='client.56363 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi162-74325-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:23.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: osdmap e115: 8 total, 8 up, 8 in 2024-04-27T01:13:23.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache", "force_nonempty":""}]: dispatch 2024-04-27T01:13:23.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3213881221' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-74307-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:23.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache", "force_nonempty":""}]: dispatch 2024-04-27T01:13:23.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: from='client.56378 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-74307-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:23.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:23.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:24.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[26714]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache", "force_nonempty":""}]': finished 2024-04-27T01:13:24.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[26714]: from='client.56378 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-74307-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:24.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74627-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:24.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[26714]: osdmap e116: 8 total, 8 up, 8 in 2024-04-27T01:13:24.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[26714]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74627-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:24.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:24.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[26714]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:24.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[37680]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache", "force_nonempty":""}]': finished 2024-04-27T01:13:24.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[37680]: from='client.56378 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-74307-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:24.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74627-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:24.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[37680]: osdmap e116: 8 total, 8 up, 8 in 2024-04-27T01:13:24.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[37680]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74627-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:24.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:24.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:24 smithi162 ceph-mon[37680]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:24.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:24 smithi181 ceph-mon[30237]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache", "force_nonempty":""}]': finished 2024-04-27T01:13:24.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:24 smithi181 ceph-mon[30237]: from='client.56378 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi162-74307-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:24.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74627-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:24.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:24 smithi181 ceph-mon[30237]: osdmap e116: 8 total, 8 up, 8 in 2024-04-27T01:13:24.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:24 smithi181 ceph-mon[30237]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74627-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:24.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:24.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:24 smithi181 ceph-mon[30237]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:25.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:25 smithi181 ceph-mon[30237]: pgmap v132: 460 pgs: 49 creating+peering, 55 unknown, 356 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 256 KiB/s wr, 2 op/s 2024-04-27T01:13:25.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:25 smithi181 ceph-mon[30237]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74627-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:25.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache"}]: dispatch 2024-04-27T01:13:25.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:25 smithi181 ceph-mon[30237]: osdmap e117: 8 total, 8 up, 8 in 2024-04-27T01:13:25.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2977288815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:25 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:25 smithi181 ceph-mon[30237]: from='client.56390 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:25 smithi181 ceph-mon[30237]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache"}]: dispatch 2024-04-27T01:13:25.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:25.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[37680]: pgmap v132: 460 pgs: 49 creating+peering, 55 unknown, 356 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 256 KiB/s wr, 2 op/s 2024-04-27T01:13:25.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[37680]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74627-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:25.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache"}]: dispatch 2024-04-27T01:13:25.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[37680]: osdmap e117: 8 total, 8 up, 8 in 2024-04-27T01:13:25.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2977288815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[37680]: from='client.56390 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[37680]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache"}]: dispatch 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[26714]: pgmap v132: 460 pgs: 49 creating+peering, 55 unknown, 356 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 256 KiB/s wr, 2 op/s 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[26714]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74627-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1149217675' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache"}]: dispatch 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[26714]: osdmap e117: 8 total, 8 up, 8 in 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2977288815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[26714]: from='client.56390 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:25.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[26714]: from='client.48649 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache"}]: dispatch 2024-04-27T01:13:25.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:26.108 INFO:tasks.workunit.client.0.smithi162.stdout: misc: Running main() from gmock_main.cc 2024-04-27T01:13:26.108 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-04-27T01:13:26.108 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [----------] Global test environment set-up. 2024-04-27T01:13:26.108 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-04-27T01:13:26.108 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-04-27T01:13:26.108 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.Version (9232 ms) 2024-04-27T01:13:26.108 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-04-27T01:13:26.108 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (7397 ms) 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.LongName (3334 ms) 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3624 ms) 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3350 ms) 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4472 ms) 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.Exec (3648 ms) 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (4325 ms) 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-04-27T01:13:26.109 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3807 ms) 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3821 ms) 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (3016 ms) 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3006 ms) 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [----------] 12 tests from NeoRadosMisc (53032 ms total) 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [----------] Global test environment tear-down 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [==========] 12 tests from 1 test suite ran. (53032 ms total) 2024-04-27T01:13:26.110 INFO:tasks.workunit.client.0.smithi162.stdout: misc: [ PASSED ] 12 tests. 2024-04-27T01:13:26.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:26 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:26.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:26 smithi181 ceph-mon[30237]: from='client.56390 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:26.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:26 smithi181 ceph-mon[30237]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache"}]': finished 2024-04-27T01:13:26.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:26 smithi181 ceph-mon[30237]: osdmap e118: 8 total, 8 up, 8 in 2024-04-27T01:13:26.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:26 smithi181 ceph-mon[30237]: pgmap v135: 524 pgs: 23 creating+peering, 145 unknown, 356 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 3 op/s 2024-04-27T01:13:26.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:26.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:26 smithi181 ceph-mon[30237]: osdmap e119: 8 total, 8 up, 8 in 2024-04-27T01:13:26.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:26.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[37680]: from='client.56390 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:26.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[37680]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache"}]': finished 2024-04-27T01:13:26.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[37680]: osdmap e118: 8 total, 8 up, 8 in 2024-04-27T01:13:26.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[37680]: pgmap v135: 524 pgs: 23 creating+peering, 145 unknown, 356 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 3 op/s 2024-04-27T01:13:26.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:26.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[37680]: osdmap e119: 8 total, 8 up, 8 in 2024-04-27T01:13:26.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:26.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[26714]: from='client.56390 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:26.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[26714]: from='client.48649 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74627-10", "tierpool":"test-rados-api-smithi162-74627-10-cache"}]': finished 2024-04-27T01:13:26.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[26714]: osdmap e118: 8 total, 8 up, 8 in 2024-04-27T01:13:26.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[26714]: pgmap v135: 524 pgs: 23 creating+peering, 145 unknown, 356 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 3 op/s 2024-04-27T01:13:26.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:26.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:26 smithi162 ceph-mon[26714]: osdmap e119: 8 total, 8 up, 8 in 2024-04-27T01:13:27.102 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: Running main() from gmock_main.cc 2024-04-27T01:13:27.102 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-04-27T01:13:27.102 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [----------] Global test environment set-up. 2024-04-27T01:13:27.102 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-04-27T01:13:27.102 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolList (10444 ms) 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (7506 ms) 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3103 ms) 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3947 ms) 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3376 ms) 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6978 ms) 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5373 ms) 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5590 ms) 2024-04-27T01:13:27.103 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-04-27T01:13:27.104 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8022 ms) 2024-04-27T01:13:27.104 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [----------] 9 tests from LibRadosPools (54339 ms total) 2024-04-27T01:13:27.104 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: 2024-04-27T01:13:27.104 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [----------] Global test environment tear-down 2024-04-27T01:13:27.104 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (54339 ms total) 2024-04-27T01:13:27.104 INFO:tasks.workunit.client.0.smithi162.stdout: api_pool: [ PASSED ] 9 tests. 2024-04-27T01:13:27.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3643718125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-74307-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-74307-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:27.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:27.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:27.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-74307-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:27.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:27.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: osdmap e120: 8 total, 8 up, 8 in 2024-04-27T01:13:27.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:27.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4001642541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-74325-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:27.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:27 smithi181 ceph-mon[30237]: from='client.56414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-74325-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3643718125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-74307-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-74307-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:27.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:27.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:27.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-74307-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:27.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:27.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: osdmap e120: 8 total, 8 up, 8 in 2024-04-27T01:13:27.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4001642541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-74325-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[37680]: from='client.56414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-74325-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3643718125' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-74307-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.48691 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-74307-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.48691 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi162-74307-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:27.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: osdmap e120: 8 total, 8 up, 8 in 2024-04-27T01:13:27.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:27.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4001642541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-74325-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:27.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:27.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:27 smithi162 ceph-mon[26714]: from='client.56414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-74325-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:28 smithi181 ceph-mon[30237]: pgmap v138: 460 pgs: 46 creating+peering, 101 unknown, 313 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:13:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-11"}]': finished 2024-04-27T01:13:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:28 smithi181 ceph-mon[30237]: from='client.56414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-74325-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-11", "mode": "writeback"}]: dispatch 2024-04-27T01:13:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:28 smithi181 ceph-mon[30237]: osdmap e121: 8 total, 8 up, 8 in 2024-04-27T01:13:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-11", "mode": "writeback"}]: dispatch 2024-04-27T01:13:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[37680]: pgmap v138: 460 pgs: 46 creating+peering, 101 unknown, 313 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:13:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-11"}]': finished 2024-04-27T01:13:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[37680]: from='client.56414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-74325-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-11", "mode": "writeback"}]: dispatch 2024-04-27T01:13:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[37680]: osdmap e121: 8 total, 8 up, 8 in 2024-04-27T01:13:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-11", "mode": "writeback"}]: dispatch 2024-04-27T01:13:28.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[26714]: pgmap v138: 460 pgs: 46 creating+peering, 101 unknown, 313 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:13:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-11"}]': finished 2024-04-27T01:13:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[26714]: from='client.56414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi162-74325-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-11", "mode": "writeback"}]: dispatch 2024-04-27T01:13:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[26714]: osdmap e121: 8 total, 8 up, 8 in 2024-04-27T01:13:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-11", "mode": "writeback"}]: dispatch 2024-04-27T01:13:29.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:29.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:29.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-11", "mode": "writeback"}]': finished 2024-04-27T01:13:29.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[37680]: osdmap e122: 8 total, 8 up, 8 in 2024-04-27T01:13:29.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/763576505' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-74307-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:29.731 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[37680]: from='client.56429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-74307-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:29.731 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:29.731 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:29.731 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-11", "mode": "writeback"}]': finished 2024-04-27T01:13:29.731 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[26714]: osdmap e122: 8 total, 8 up, 8 in 2024-04-27T01:13:29.731 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/763576505' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-74307-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:29.731 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-mon[26714]: from='client.56429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-74307-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:29.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:29.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:29 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:29.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:29 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-11", "mode": "writeback"}]': finished 2024-04-27T01:13:29.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:29 smithi181 ceph-mon[30237]: osdmap e122: 8 total, 8 up, 8 in 2024-04-27T01:13:29.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/763576505' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-74307-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:29.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:29 smithi181 ceph-mon[30237]: from='client.56429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-74307-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:30.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:13:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:13:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:13:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:30 smithi181 ceph-mon[30237]: pgmap v141: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:13:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:30 smithi181 ceph-mon[30237]: from='client.56429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-74307-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:30 smithi181 ceph-mon[30237]: osdmap e123: 8 total, 8 up, 8 in 2024-04-27T01:13:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2684107477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-74325-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:30 smithi181 ceph-mon[30237]: from='client.56438 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-74325-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:30.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:30.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:30 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[26714]: pgmap v141: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:13:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[26714]: from='client.56429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-74307-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[26714]: osdmap e123: 8 total, 8 up, 8 in 2024-04-27T01:13:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2684107477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-74325-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[26714]: from='client.56438 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-74325-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:30.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[37680]: pgmap v141: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:13:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[37680]: from='client.56429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi162-74307-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[37680]: osdmap e123: 8 total, 8 up, 8 in 2024-04-27T01:13:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2684107477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-74325-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[37680]: from='client.56438 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-74325-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:30 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:13:32.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:32.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:31 smithi181 ceph-mon[30237]: from='client.56438 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-74325-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:32.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:31 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:13:32.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:31 smithi181 ceph-mon[30237]: osdmap e124: 8 total, 8 up, 8 in 2024-04-27T01:13:32.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:32.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:31 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:32.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:32.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[37680]: from='client.56438 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-74325-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:32.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:13:32.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[37680]: osdmap e124: 8 total, 8 up, 8 in 2024-04-27T01:13:32.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:32.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:32.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:32.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[26714]: from='client.56438 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi162-74325-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:32.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:13:32.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[26714]: osdmap e124: 8 total, 8 up, 8 in 2024-04-27T01:13:32.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:32.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:31 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11"}]: dispatch 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: Running main() from gmock_main.cc 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [----------] Global test environment set-up. 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [----------] 14 tests from NeoRadosIo 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.Limits (11074 ms) 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (4358 ms) 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.ReadOp (4557 ms) 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.SparseRead (3613 ms) 2024-04-27T01:13:32.103 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.RoundTrip (4475 ms) 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3367 ms) 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3607 ms) 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3321 ms) 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3850 ms) 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.Trunc (3249 ms) 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.Remove (4575 ms) 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3020 ms) 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-04-27T01:13:32.104 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.RmXattr (2987 ms) 2024-04-27T01:13:32.105 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-04-27T01:13:32.105 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3006 ms) 2024-04-27T01:13:32.105 INFO:tasks.workunit.client.0.smithi162.stdout: io: [----------] 14 tests from NeoRadosIo (59059 ms total) 2024-04-27T01:13:32.105 INFO:tasks.workunit.client.0.smithi162.stdout: io: 2024-04-27T01:13:32.105 INFO:tasks.workunit.client.0.smithi162.stdout: io: [----------] Global test environment tear-down 2024-04-27T01:13:32.105 INFO:tasks.workunit.client.0.smithi162.stdout: io: [==========] 14 tests from 1 test suite ran. (59059 ms total) 2024-04-27T01:13:32.105 INFO:tasks.workunit.client.0.smithi162.stdout: io: [ PASSED ] 14 tests. 2024-04-27T01:13:32.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:13:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:13:33.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:32 smithi181 ceph-mon[30237]: pgmap v144: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-04-27T01:13:33.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:33.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:32 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:13:33.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:32 smithi181 ceph-mon[30237]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:33.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:32 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11"}]': finished 2024-04-27T01:13:33.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:32 smithi181 ceph-mon[30237]: osdmap e125: 8 total, 8 up, 8 in 2024-04-27T01:13:33.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/32400860' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-74307-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:33.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:32 smithi181 ceph-mon[30237]: from='client.56450 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-74307-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:33.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[37680]: pgmap v144: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-04-27T01:13:33.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:33.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:13:33.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[37680]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:33.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11"}]': finished 2024-04-27T01:13:33.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[37680]: osdmap e125: 8 total, 8 up, 8 in 2024-04-27T01:13:33.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/32400860' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-74307-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:33.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[37680]: from='client.56450 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-74307-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:33.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[26714]: pgmap v144: 428 pgs: 96 unknown, 332 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-04-27T01:13:33.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:33.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:13:33.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[26714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:33.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-11"}]': finished 2024-04-27T01:13:33.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[26714]: osdmap e125: 8 total, 8 up, 8 in 2024-04-27T01:13:33.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/32400860' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-74307-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:33.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:32 smithi162 ceph-mon[26714]: from='client.56450 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-74307-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:34.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:34.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:34.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:33 smithi181 ceph-mon[30237]: from='client.56450 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-74307-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:34.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:33 smithi181 ceph-mon[30237]: osdmap e126: 8 total, 8 up, 8 in 2024-04-27T01:13:34.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:34.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1392319261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-74325-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:34.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:33 smithi181 ceph-mon[30237]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:34.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:33 smithi181 ceph-mon[30237]: from='client.48736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-74325-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:34.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:34.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:34.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[37680]: from='client.56450 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-74307-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:34.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[37680]: osdmap e126: 8 total, 8 up, 8 in 2024-04-27T01:13:34.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:34.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1392319261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-74325-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:34.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[37680]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:34.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[37680]: from='client.48736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-74325-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:34.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:34.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:34.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[26714]: from='client.56450 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi162-74307-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:34.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[26714]: osdmap e126: 8 total, 8 up, 8 in 2024-04-27T01:13:34.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:34.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1392319261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-74325-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:34.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[26714]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:34.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:33 smithi162 ceph-mon[26714]: from='client.48736 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-74325-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:34.119 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: Running main() from gmock_main.cc 2024-04-27T01:13:34.119 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-04-27T01:13:34.119 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [----------] Global test environment set-up. 2024-04-27T01:13:34.119 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-04-27T01:13:34.119 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-04-27T01:13:34.119 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (11020 ms) 2024-04-27T01:13:34.119 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-04-27T01:13:34.119 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (4365 ms) 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (4557 ms) 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3616 ms) 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (4468 ms) 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (4663 ms) 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (4468 ms) 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3879 ms) 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-04-27T01:13:34.120 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (4356 ms) 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (2853 ms) 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3767 ms) 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3007 ms) 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2986 ms) 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3023 ms) 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (61029 ms total) 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [----------] Global test environment tear-down 2024-04-27T01:13:34.121 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (61029 ms total) 2024-04-27T01:13:34.122 INFO:tasks.workunit.client.0.smithi162.stdout: read_operations: [ PASSED ] 14 tests. 2024-04-27T01:13:35.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:34 smithi181 ceph-mon[30237]: pgmap v147: 420 pgs: 33 creating+peering, 63 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-27T01:13:35.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:35.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:34 smithi181 ceph-mon[30237]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]': finished 2024-04-27T01:13:35.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:34 smithi181 ceph-mon[30237]: from='client.48736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-74325-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:35.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:34 smithi181 ceph-mon[30237]: osdmap e127: 8 total, 8 up, 8 in 2024-04-27T01:13:35.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:35.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:34 smithi181 ceph-mon[30237]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:35.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:35.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:35.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[37680]: pgmap v147: 420 pgs: 33 creating+peering, 63 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-27T01:13:35.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:35.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[37680]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]': finished 2024-04-27T01:13:35.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[37680]: from='client.48736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-74325-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:35.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[37680]: osdmap e127: 8 total, 8 up, 8 in 2024-04-27T01:13:35.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:35.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[37680]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:35.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:35.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:35.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[26714]: pgmap v147: 420 pgs: 33 creating+peering, 63 unknown, 324 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.7 KiB/s wr, 6 op/s 2024-04-27T01:13:35.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:35.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[26714]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi162-74726-10"}]': finished 2024-04-27T01:13:35.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[26714]: from='client.48736 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi162-74325-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:35.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[26714]: osdmap e127: 8 total, 8 up, 8 in 2024-04-27T01:13:35.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/199686394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:35.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[26714]: from='client.48511 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]: dispatch 2024-04-27T01:13:35.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:35.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:36.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:36.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:35 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:36.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:35 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:36.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:35 smithi181 ceph-mon[30237]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]': finished 2024-04-27T01:13:36.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:35 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:36.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:35 smithi181 ceph-mon[30237]: osdmap e128: 8 total, 8 up, 8 in 2024-04-27T01:13:36.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1082357534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi162-74307-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:36.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:36.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:36.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:36.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:36.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[26714]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]': finished 2024-04-27T01:13:36.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:36.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[26714]: osdmap e128: 8 total, 8 up, 8 in 2024-04-27T01:13:36.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1082357534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi162-74307-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:36.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:36.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:36.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:36.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:36.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[37680]: from='client.48511 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi162-74726-10"}]': finished 2024-04-27T01:13:36.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:36.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[37680]: osdmap e128: 8 total, 8 up, 8 in 2024-04-27T01:13:36.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1082357534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi162-74307-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:36.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: pgmap v150: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:13:37.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:37.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:37.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:37.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1082357534' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi162-74307-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:37.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: osdmap e129: 8 total, 8 up, 8 in 2024-04-27T01:13:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2808685696' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-74325-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:37.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: pgmap v150: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:13:37.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:37.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:37.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:37.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1082357534' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi162-74307-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: osdmap e129: 8 total, 8 up, 8 in 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2808685696' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-74325-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: pgmap v150: 356 pgs: 64 unknown, 292 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:37.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1082357534' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi162-74307-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:37.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:37.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: osdmap e129: 8 total, 8 up, 8 in 2024-04-27T01:13:37.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:37.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2808685696' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-74325-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:38.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:38.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2808685696' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-74325-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:38.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:37 smithi181 ceph-mon[30237]: osdmap e130: 8 total, 8 up, 8 in 2024-04-27T01:13:38.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:38.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2808685696' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-74325-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:38.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:37 smithi162 ceph-mon[26714]: osdmap e130: 8 total, 8 up, 8 in 2024-04-27T01:13:38.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:38.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2808685696' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi162-74325-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:38.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:37 smithi162 ceph-mon[37680]: osdmap e130: 8 total, 8 up, 8 in 2024-04-27T01:13:39.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:38 smithi181 ceph-mon[30237]: pgmap v153: 388 pgs: 46 creating+peering, 29 unknown, 313 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 28 KiB/s wr, 28 op/s 2024-04-27T01:13:39.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:39.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:38 smithi181 ceph-mon[30237]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]': finished 2024-04-27T01:13:39.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:38 smithi181 ceph-mon[30237]: osdmap e131: 8 total, 8 up, 8 in 2024-04-27T01:13:39.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/245347780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-74307-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:39.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:38 smithi181 ceph-mon[30237]: from='client.48757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-74307-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:39.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[37680]: pgmap v153: 388 pgs: 46 creating+peering, 29 unknown, 313 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 28 KiB/s wr, 28 op/s 2024-04-27T01:13:39.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:39.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[37680]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]': finished 2024-04-27T01:13:39.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[37680]: osdmap e131: 8 total, 8 up, 8 in 2024-04-27T01:13:39.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/245347780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-74307-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:39.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[37680]: from='client.48757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-74307-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:39.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[26714]: pgmap v153: 388 pgs: 46 creating+peering, 29 unknown, 313 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 28 KiB/s wr, 28 op/s 2024-04-27T01:13:39.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:39.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[26714]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi162-74726-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]': finished 2024-04-27T01:13:39.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[26714]: osdmap e131: 8 total, 8 up, 8 in 2024-04-27T01:13:39.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/245347780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-74307-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:39.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:38 smithi162 ceph-mon[26714]: from='client.48757 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-74307-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:40.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:40.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:39 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:40.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:39 smithi181 ceph-mon[30237]: from='client.48757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-74307-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:40.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:39 smithi181 ceph-mon[30237]: osdmap e132: 8 total, 8 up, 8 in 2024-04-27T01:13:40.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/246513571' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-74325-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:40.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:39 smithi181 ceph-mon[30237]: from='client.56486 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-74325-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:40.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:40.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:40.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:13:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:13:40.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:40.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:40.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[37680]: from='client.48757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-74307-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:40.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[37680]: osdmap e132: 8 total, 8 up, 8 in 2024-04-27T01:13:40.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/246513571' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-74325-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:40.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[37680]: from='client.56486 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-74325-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:40.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:40.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:40.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:40.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:40.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[26714]: from='client.48757 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi162-74307-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:40.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[26714]: osdmap e132: 8 total, 8 up, 8 in 2024-04-27T01:13:40.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/246513571' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-74325-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:40.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[26714]: from='client.56486 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-74325-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:40.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:40.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:13:40.989 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[37680]: pgmap v156: 428 pgs: 4 creating+activating, 28 creating+peering, 72 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-04-27T01:13:40.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:40.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[37680]: from='client.56486 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-74325-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:40.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:40.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:13:40.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[37680]: osdmap e133: 8 total, 8 up, 8 in 2024-04-27T01:13:40.990 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:13:40.990 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[26714]: pgmap v156: 428 pgs: 4 creating+activating, 28 creating+peering, 72 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-04-27T01:13:40.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:40.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[26714]: from='client.56486 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-74325-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:40.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:40.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:13:40.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[26714]: osdmap e133: 8 total, 8 up, 8 in 2024-04-27T01:13:40.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:13:41.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:40 smithi181 ceph-mon[30237]: pgmap v156: 428 pgs: 4 creating+activating, 28 creating+peering, 72 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-04-27T01:13:41.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:41.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:40 smithi181 ceph-mon[30237]: from='client.56486 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi162-74325-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:41.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:13:41.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:13:41.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:40 smithi181 ceph-mon[30237]: osdmap e133: 8 total, 8 up, 8 in 2024-04-27T01:13:41.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:13:42.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:42.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:41 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-13"}]': finished 2024-04-27T01:13:42.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-13", "mode": "writeback"}]: dispatch 2024-04-27T01:13:42.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:41 smithi181 ceph-mon[30237]: osdmap e134: 8 total, 8 up, 8 in 2024-04-27T01:13:42.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:41 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-13", "mode": "writeback"}]: dispatch 2024-04-27T01:13:42.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2429807486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-74307-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:42.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:41 smithi181 ceph-mon[30237]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-74307-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:42.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:42.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-13"}]': finished 2024-04-27T01:13:42.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-13", "mode": "writeback"}]: dispatch 2024-04-27T01:13:42.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[37680]: osdmap e134: 8 total, 8 up, 8 in 2024-04-27T01:13:42.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-13", "mode": "writeback"}]: dispatch 2024-04-27T01:13:42.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2429807486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-74307-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:42.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[37680]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-74307-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-13"}]': finished 2024-04-27T01:13:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-13", "mode": "writeback"}]: dispatch 2024-04-27T01:13:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[26714]: osdmap e134: 8 total, 8 up, 8 in 2024-04-27T01:13:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-13", "mode": "writeback"}]: dispatch 2024-04-27T01:13:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2429807486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-74307-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:42.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:41 smithi162 ceph-mon[26714]: from='client.48769 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-74307-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:42.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:13:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:13:43.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:42 smithi181 ceph-mon[30237]: pgmap v159: 364 pgs: 40 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s wr, 17 op/s 2024-04-27T01:13:43.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:43.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:42 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:43.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:42 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-13", "mode": "writeback"}]': finished 2024-04-27T01:13:43.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:42 smithi181 ceph-mon[30237]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-74307-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:43.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:42 smithi181 ceph-mon[30237]: osdmap e135: 8 total, 8 up, 8 in 2024-04-27T01:13:43.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1845898186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-74325-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:43.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[37680]: pgmap v159: 364 pgs: 40 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s wr, 17 op/s 2024-04-27T01:13:43.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:43.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:43.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-13", "mode": "writeback"}]': finished 2024-04-27T01:13:43.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[37680]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-74307-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:43.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[37680]: osdmap e135: 8 total, 8 up, 8 in 2024-04-27T01:13:43.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1845898186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-74325-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:43.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[26714]: pgmap v159: 364 pgs: 40 unknown, 324 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s wr, 17 op/s 2024-04-27T01:13:43.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:43.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:13:43.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-13", "mode": "writeback"}]': finished 2024-04-27T01:13:43.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[26714]: from='client.48769 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi162-74307-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:43.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[26714]: osdmap e135: 8 total, 8 up, 8 in 2024-04-27T01:13:43.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1845898186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-74325-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:44.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:44.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:44.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.0"}]: dispatch 2024-04-27T01:13:44.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.0"}]: dispatch 2024-04-27T01:13:44.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.1"}]: dispatch 2024-04-27T01:13:44.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.1"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.2"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.2"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.3"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.3"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.4"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.4"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.5"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.5"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.6"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.6"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.7"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.7"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.8"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.8"}]: dispatch 2024-04-27T01:13:44.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.9"}]: dispatch 2024-04-27T01:13:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.9"}]: dispatch 2024-04-27T01:13:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: 175.4 scrub starts 2024-04-27T01:13:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: 175.4 scrub ok 2024-04-27T01:13:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1845898186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-74325-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: osdmap e136: 8 total, 8 up, 8 in 2024-04-27T01:13:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: 175.9 scrub starts 2024-04-27T01:13:44.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:43 smithi181 ceph-mon[30237]: 175.9 scrub ok 2024-04-27T01:13:44.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:44.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:44.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.0"}]: dispatch 2024-04-27T01:13:44.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.0"}]: dispatch 2024-04-27T01:13:44.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.1"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.1"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.2"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.2"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.3"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.3"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.4"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.4"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.5"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.5"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.6"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.6"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.7"}]: dispatch 2024-04-27T01:13:44.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.7"}]: dispatch 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.8"}]: dispatch 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.8"}]: dispatch 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.9"}]: dispatch 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.9"}]: dispatch 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: 175.4 scrub starts 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: 175.4 scrub ok 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1845898186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-74325-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: osdmap e136: 8 total, 8 up, 8 in 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: 175.9 scrub starts 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[37680]: 175.9 scrub ok 2024-04-27T01:13:44.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.0"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.0"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.1"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.1"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.2"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.2"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.3"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.3"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.4"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.4"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.5"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.5"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.6"}]: dispatch 2024-04-27T01:13:44.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.6"}]: dispatch 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.7"}]: dispatch 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.7"}]: dispatch 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.8"}]: dispatch 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.8"}]: dispatch 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "175.9"}]: dispatch 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "175.9"}]: dispatch 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: 175.4 scrub starts 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: 175.4 scrub ok 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1845898186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi162-74325-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: osdmap e136: 8 total, 8 up, 8 in 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: 175.9 scrub starts 2024-04-27T01:13:44.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:43 smithi162 ceph-mon[26714]: 175.9 scrub ok 2024-04-27T01:13:45.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: pgmap v162: 396 pgs: 26 creating+peering, 38 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 79 KiB/s wr, 77 op/s 2024-04-27T01:13:45.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: 175.2 scrub starts 2024-04-27T01:13:45.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: 175.2 scrub ok 2024-04-27T01:13:45.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: 175.5 scrub starts 2024-04-27T01:13:45.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: 175.5 scrub ok 2024-04-27T01:13:45.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: 175.8 scrub starts 2024-04-27T01:13:45.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: 175.8 scrub ok 2024-04-27T01:13:45.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:45.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: 175.3 scrub starts 2024-04-27T01:13:45.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: 175.3 scrub ok 2024-04-27T01:13:45.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:45.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: osdmap e137: 8 total, 8 up, 8 in 2024-04-27T01:13:45.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3094422029' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-74307-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:45.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:44 smithi181 ceph-mon[30237]: from='client.56510 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-74307-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:45.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: pgmap v162: 396 pgs: 26 creating+peering, 38 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 79 KiB/s wr, 77 op/s 2024-04-27T01:13:45.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: 175.2 scrub starts 2024-04-27T01:13:45.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: 175.2 scrub ok 2024-04-27T01:13:45.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: 175.5 scrub starts 2024-04-27T01:13:45.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: 175.5 scrub ok 2024-04-27T01:13:45.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: 175.8 scrub starts 2024-04-27T01:13:45.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: 175.8 scrub ok 2024-04-27T01:13:45.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:45.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: 175.3 scrub starts 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: 175.3 scrub ok 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: osdmap e137: 8 total, 8 up, 8 in 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3094422029' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-74307-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[37680]: from='client.56510 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-74307-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: pgmap v162: 396 pgs: 26 creating+peering, 38 unknown, 332 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 79 KiB/s wr, 77 op/s 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: 175.2 scrub starts 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: 175.2 scrub ok 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: 175.5 scrub starts 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: 175.5 scrub ok 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: 175.8 scrub starts 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: 175.8 scrub ok 2024-04-27T01:13:45.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:45.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: 175.3 scrub starts 2024-04-27T01:13:45.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: 175.3 scrub ok 2024-04-27T01:13:45.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:45.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: osdmap e137: 8 total, 8 up, 8 in 2024-04-27T01:13:45.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3094422029' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-74307-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:45.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:44 smithi162 ceph-mon[26714]: from='client.56510 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-74307-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: 175.6 scrub starts 2024-04-27T01:13:46.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: 175.6 scrub ok 2024-04-27T01:13:46.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: 175.1 scrub starts 2024-04-27T01:13:46.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: 175.1 scrub ok 2024-04-27T01:13:46.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:46.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: 175.0 scrub starts 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: 175.0 scrub ok 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.56510 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-74307-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: osdmap e138: 8 total, 8 up, 8 in 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2"}]: dispatch 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2016748430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-74325-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.48781 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-74325-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:45 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: 175.6 scrub starts 2024-04-27T01:13:46.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: 175.6 scrub ok 2024-04-27T01:13:46.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: 175.1 scrub starts 2024-04-27T01:13:46.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: 175.1 scrub ok 2024-04-27T01:13:46.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: 175.0 scrub starts 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: 175.0 scrub ok 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.56510 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-74307-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: osdmap e138: 8 total, 8 up, 8 in 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2"}]: dispatch 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2016748430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-74325-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.48781 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-74325-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: 175.6 scrub starts 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: 175.6 scrub ok 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: 175.1 scrub starts 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: 175.1 scrub ok 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: 175.0 scrub starts 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: 175.0 scrub ok 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.56510 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi162-74307-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: osdmap e138: 8 total, 8 up, 8 in 2024-04-27T01:13:46.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2"}]: dispatch 2024-04-27T01:13:46.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2016748430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-74325-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.48781 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-74325-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:46.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:45 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:47.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: pgmap v165: 364 pgs: 64 unknown, 300 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 68 op/s 2024-04-27T01:13:47.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: 175.7 scrub starts 2024-04-27T01:13:47.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: 175.7 scrub ok 2024-04-27T01:13:47.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:47.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: from='client.48781 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-74325-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:47.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:47.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: osdmap e139: 8 total, 8 up, 8 in 2024-04-27T01:13:47.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:13:47.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4129411670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74682-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:47.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:13:47.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:13:47.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:46 smithi181 ceph-mon[30237]: from='client.56525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74682-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:47.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: pgmap v165: 364 pgs: 64 unknown, 300 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 68 op/s 2024-04-27T01:13:47.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: 175.7 scrub starts 2024-04-27T01:13:47.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: 175.7 scrub ok 2024-04-27T01:13:47.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:47.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: from='client.48781 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-74325-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:47.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:47.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: osdmap e139: 8 total, 8 up, 8 in 2024-04-27T01:13:47.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:13:47.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4129411670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74682-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:47.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:13:47.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:13:47.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[37680]: from='client.56525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74682-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:47.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: pgmap v165: 364 pgs: 64 unknown, 300 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 68 op/s 2024-04-27T01:13:47.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: 175.7 scrub starts 2024-04-27T01:13:47.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: 175.7 scrub ok 2024-04-27T01:13:47.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:47.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: from='client.48781 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi162-74325-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:47.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi162-74610-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:47.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: osdmap e139: 8 total, 8 up, 8 in 2024-04-27T01:13:47.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:13:47.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4129411670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74682-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:47.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:13:47.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-27T01:13:47.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:46 smithi162 ceph-mon[26714]: from='client.56525 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74682-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:48.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:48.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:47 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-27T01:13:48.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:47 smithi181 ceph-mon[30237]: from='client.56525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74682-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:48.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:47 smithi181 ceph-mon[30237]: osdmap e140: 8 total, 8 up, 8 in 2024-04-27T01:13:48.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:13:48.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:48.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2989490026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-74307-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:48.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:47 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:13:48.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:47 smithi181 ceph-mon[30237]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:48.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:47 smithi181 ceph-mon[30237]: from='client.56531 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-74307-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:48.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:48.144 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-27T01:13:48.144 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[37680]: from='client.56525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74682-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:48.144 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[37680]: osdmap e140: 8 total, 8 up, 8 in 2024-04-27T01:13:48.144 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2989490026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-74307-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[37680]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[37680]: from='client.56531 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-74307-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[26714]: from='client.56525 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74682-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[26714]: osdmap e140: 8 total, 8 up, 8 in 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:13:48.145 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:48.146 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2989490026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-74307-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:48.146 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-27T01:13:48.146 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[26714]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:48.146 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:47 smithi162 ceph-mon[26714]: from='client.56531 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-74307-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:49.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: pgmap v168: 388 pgs: 37 creating+peering, 59 unknown, 292 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 57 op/s 2024-04-27T01:13:49.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:49.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-27T01:13:49.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]': finished 2024-04-27T01:13:49.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: from='client.56531 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-74307-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:49.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: osdmap e141: 8 total, 8 up, 8 in 2024-04-27T01:13:49.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:49.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:13:49.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3712010047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:49.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:49.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:13:49.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:48 smithi181 ceph-mon[30237]: from='client.56543 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:49.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: pgmap v168: 388 pgs: 37 creating+peering, 59 unknown, 292 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 57 op/s 2024-04-27T01:13:49.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:49.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-27T01:13:49.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]': finished 2024-04-27T01:13:49.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: from='client.56531 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-74307-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:49.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: osdmap e141: 8 total, 8 up, 8 in 2024-04-27T01:13:49.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:49.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:13:49.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3712010047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:49.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:49.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:13:49.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[37680]: from='client.56543 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:49.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: pgmap v168: 388 pgs: 37 creating+peering, 59 unknown, 292 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 57 op/s 2024-04-27T01:13:49.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:49.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-27T01:13:49.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]': finished 2024-04-27T01:13:49.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: from='client.56531 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi162-74307-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:49.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: osdmap e141: 8 total, 8 up, 8 in 2024-04-27T01:13:49.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2964381677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:49.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/745107674' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:13:49.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3712010047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:49.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: from='client.56474 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]: dispatch 2024-04-27T01:13:49.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1"}]: dispatch 2024-04-27T01:13:49.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:48 smithi162 ceph-mon[26714]: from='client.56543 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:49.138 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: Running main() from gmock_main.cc 2024-04-27T01:13:49.138 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-04-27T01:13:49.138 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] Global test environment set-up. 2024-04-27T01:13:49.138 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-04-27T01:13:49.138 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-04-27T01:13:49.138 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (4839 ms) 2024-04-27T01:13:49.138 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-04-27T01:13:49.139 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (4219 ms) 2024-04-27T01:13:49.139 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-04-27T01:13:49.139 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2317 ms) 2024-04-27T01:13:49.139 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-04-27T01:13:49.139 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2914 ms) 2024-04-27T01:13:49.139 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (14290 ms total) 2024-04-27T01:13:49.139 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: 2024-04-27T01:13:49.139 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-04-27T01:13:49.139 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-04-27T01:13:49.140 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (5870 ms) 2024-04-27T01:13:49.140 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-04-27T01:13:49.140 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4479 ms) 2024-04-27T01:13:49.140 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-04-27T01:13:49.140 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (7206 ms) 2024-04-27T01:13:49.140 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (17555 ms total) 2024-04-27T01:13:49.140 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: 2024-04-27T01:13:49.140 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-04-27T01:13:49.140 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-04-27T01:13:49.141 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (4023 ms) 2024-04-27T01:13:49.141 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-04-27T01:13:49.141 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1998 ms) 2024-04-27T01:13:49.141 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-04-27T01:13:49.141 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1987 ms) 2024-04-27T01:13:49.141 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-04-27T01:13:49.141 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2005 ms) 2024-04-27T01:13:49.141 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (10013 ms total) 2024-04-27T01:13:49.142 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: 2024-04-27T01:13:49.142 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-04-27T01:13:49.142 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-04-27T01:13:49.142 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4007 ms) 2024-04-27T01:13:49.142 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-04-27T01:13:49.142 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4015 ms) 2024-04-27T01:13:49.142 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8022 ms total) 2024-04-27T01:13:49.142 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: 2024-04-27T01:13:49.142 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [----------] Global test environment tear-down 2024-04-27T01:13:49.143 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (76321 ms total) 2024-04-27T01:13:49.143 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-04-27T01:13:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:49 smithi181 ceph-mon[30237]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]': finished 2024-04-27T01:13:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:49 smithi181 ceph-mon[30237]: from='client.24721 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1"}]': finished 2024-04-27T01:13:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:49 smithi181 ceph-mon[30237]: from='client.56543 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:50.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:49 smithi181 ceph-mon[30237]: osdmap e142: 8 total, 8 up, 8 in 2024-04-27T01:13:50.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:13:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:13:50.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:50.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-mon[37680]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]': finished 2024-04-27T01:13:50.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-mon[37680]: from='client.24721 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1"}]': finished 2024-04-27T01:13:50.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-mon[37680]: from='client.56543 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:50.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-mon[37680]: osdmap e142: 8 total, 8 up, 8 in 2024-04-27T01:13:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-mon[26714]: from='client.56474 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi162-74726-15"}]': finished 2024-04-27T01:13:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-mon[26714]: from='client.24721 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi162-74610-1","app":"app1","key":"key1"}]': finished 2024-04-27T01:13:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-mon[26714]: from='client.56543 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:50.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:49 smithi162 ceph-mon[26714]: osdmap e142: 8 total, 8 up, 8 in 2024-04-27T01:13:50.165 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-04-27T01:13:50.165 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-04-27T01:13:50.165 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-04-27T01:13:50.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-04-27T01:13:50.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-04-27T01:13:50.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-27T01:13:50.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-04-27T01:13:50.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-04-27T01:13:50.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: seed 74610 2024-04-27T01:13:50.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-04-27T01:13:50.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2024-04-27T01:13:50.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-04-27T01:13:50.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (391 ms) 2024-04-27T01:13:50.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-04-27T01:13:50.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (13 ms) 2024-04-27T01:13:50.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-04-27T01:13:50.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (7 ms) 2024-04-27T01:13:50.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-04-27T01:13:50.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (7 ms) 2024-04-27T01:13:50.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-04-27T01:13:50.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-04-27T01:13:50.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-04-27T01:13:50.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-04-27T01:13:50.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-04-27T01:13:50.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (9 ms) 2024-04-27T01:13:50.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-04-27T01:13:50.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-04-27T01:13:50.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-04-27T01:13:50.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (12 ms) 2024-04-27T01:13:50.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-04-27T01:13:50.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-04-27T01:13:50.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-04-27T01:13:50.170 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (8 ms) 2024-04-27T01:13:50.170 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-04-27T01:13:50.170 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (8 ms) 2024-04-27T01:13:50.170 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-04-27T01:13:50.170 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: osd_max_attr_size = 0 2024-04-27T01:13:50.170 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-04-27T01:13:50.170 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1622 ms) 2024-04-27T01:13:50.170 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-04-27T01:13:50.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (469 ms) 2024-04-27T01:13:50.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-04-27T01:13:50.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-04-27T01:13:50.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: done waiting, doing copies 2024-04-27T01:13:50.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: waiting for final deep scrubs... 2024-04-27T01:13:50.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: done waiting 2024-04-27T01:13:50.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60717 ms) 2024-04-27T01:13:50.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-04-27T01:13:50.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (5 ms) 2024-04-27T01:13:50.171 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-04-27T01:13:50.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (5 ms) 2024-04-27T01:13:50.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-04-27T01:13:50.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4201 ms) 2024-04-27T01:13:50.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-04-27T01:13:50.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-04-27T01:13:50.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-04-27T01:13:50.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-04-27T01:13:50.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-04-27T01:13:50.172 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-04-27T01:13:51.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: pgmap v171: 388 pgs: 18 creating+activating, 42 creating+peering, 32 unknown, 296 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 77 op/s 2024-04-27T01:13:51.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:51.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:51.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:51.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:51.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: osdmap e143: 8 total, 8 up, 8 in 2024-04-27T01:13:51.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4234889516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74307-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74307-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74610-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74610-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:50 smithi181 ceph-mon[30237]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: pgmap v171: 388 pgs: 18 creating+activating, 42 creating+peering, 32 unknown, 296 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 77 op/s 2024-04-27T01:13:51.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:51.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:51.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:51.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:51.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: osdmap e143: 8 total, 8 up, 8 in 2024-04-27T01:13:51.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4234889516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74307-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:51.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74307-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74610-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74610-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[37680]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: pgmap v171: 388 pgs: 18 creating+activating, 42 creating+peering, 32 unknown, 296 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 77 op/s 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: osdmap e143: 8 total, 8 up, 8 in 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4234889516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74307-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74307-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74610-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74610-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:51.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:50 smithi162 ceph-mon[26714]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:13:52.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:52.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74307-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:52.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74610-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:52.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:52.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: osdmap e144: 8 total, 8 up, 8 in 2024-04-27T01:13:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74610-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-74682-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2680980054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-74325-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74610-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-74682-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:52.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:51 smithi181 ceph-mon[30237]: from='client.48832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-74325-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:52.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:52.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74307-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:52.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74610-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:52.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:52.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: osdmap e144: 8 total, 8 up, 8 in 2024-04-27T01:13:52.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74610-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:52.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-74682-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2680980054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-74325-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74610-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-74682-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[37680]: from='client.48832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-74325-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74307-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74610-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: osdmap e144: 8 total, 8 up, 8 in 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74610-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:52.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-74682-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:52.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2680980054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-74325-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:52.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74610-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:52.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-74682-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:13:52.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:51 smithi162 ceph-mon[26714]: from='client.48832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-74325-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:52.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:13:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:13:53.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:52 smithi181 ceph-mon[30237]: pgmap v174: 356 pgs: 11 creating+activating, 19 creating+peering, 64 unknown, 262 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-27T01:13:53.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:53.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:52 smithi181 ceph-mon[30237]: from='client.48832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-74325-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:53.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:52 smithi181 ceph-mon[30237]: osdmap e145: 8 total, 8 up, 8 in 2024-04-27T01:13:53.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:52 smithi162 ceph-mon[37680]: pgmap v174: 356 pgs: 11 creating+activating, 19 creating+peering, 64 unknown, 262 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-27T01:13:53.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:53.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:52 smithi162 ceph-mon[37680]: from='client.48832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-74325-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:53.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:52 smithi162 ceph-mon[37680]: osdmap e145: 8 total, 8 up, 8 in 2024-04-27T01:13:53.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:52 smithi162 ceph-mon[26714]: pgmap v174: 356 pgs: 11 creating+activating, 19 creating+peering, 64 unknown, 262 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-27T01:13:53.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:53.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:52 smithi162 ceph-mon[26714]: from='client.48832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi162-74325-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:53.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:52 smithi162 ceph-mon[26714]: osdmap e145: 8 total, 8 up, 8 in 2024-04-27T01:13:54.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:54.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:54.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:53 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74610-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74610-24"}]': finished 2024-04-27T01:13:54.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:53 smithi181 ceph-mon[30237]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-74682-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]': finished 2024-04-27T01:13:54.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:53 smithi181 ceph-mon[30237]: osdmap e146: 8 total, 8 up, 8 in 2024-04-27T01:13:54.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2319295339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi162-74307-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:54.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:53 smithi181 ceph-mon[30237]: from='client.56591 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi162-74307-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:54.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:54.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:54.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:54.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74610-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74610-24"}]': finished 2024-04-27T01:13:54.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[26714]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-74682-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]': finished 2024-04-27T01:13:54.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[26714]: osdmap e146: 8 total, 8 up, 8 in 2024-04-27T01:13:54.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2319295339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi162-74307-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:54.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[26714]: from='client.56591 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi162-74307-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:54.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:54.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:13:54.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:54.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74610-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74610-24"}]': finished 2024-04-27T01:13:54.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[37680]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi162-74682-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]': finished 2024-04-27T01:13:54.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[37680]: osdmap e146: 8 total, 8 up, 8 in 2024-04-27T01:13:54.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2319295339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi162-74307-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:54.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[37680]: from='client.56591 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi162-74307-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:54.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:54 smithi181 ceph-mon[30237]: pgmap v176: 324 pgs: 7 creating+activating, 25 creating+peering, 292 active+clean; 592 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:13:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:54 smithi181 ceph-mon[30237]: from='client.56591 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi162-74307-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:54 smithi181 ceph-mon[30237]: osdmap e147: 8 total, 8 up, 8 in 2024-04-27T01:13:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4082993329' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-74325-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:54 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:54 smithi181 ceph-mon[30237]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-74325-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:55.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[37680]: pgmap v176: 324 pgs: 7 creating+activating, 25 creating+peering, 292 active+clean; 592 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:13:55.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[37680]: from='client.56591 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi162-74307-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:55.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[37680]: osdmap e147: 8 total, 8 up, 8 in 2024-04-27T01:13:55.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4082993329' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-74325-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[37680]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-74325-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:55.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[26714]: pgmap v176: 324 pgs: 7 creating+activating, 25 creating+peering, 292 active+clean; 592 KiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:13:55.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[26714]: from='client.56591 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi162-74307-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:55.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[26714]: osdmap e147: 8 total, 8 up, 8 in 2024-04-27T01:13:55.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4082993329' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-74325-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[26714]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-74325-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:55.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:56.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:56 smithi181 ceph-mon[30237]: pgmap v179: 404 pgs: 112 unknown, 292 active+clean; 592 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:13:56.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:56 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:56.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:56 smithi181 ceph-mon[30237]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-74325-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:56.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:56 smithi181 ceph-mon[30237]: osdmap e148: 8 total, 8 up, 8 in 2024-04-27T01:13:56.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:56.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:56 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:56.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:56.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:56 smithi181 ceph-mon[30237]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[37680]: pgmap v179: 404 pgs: 112 unknown, 292 active+clean; 592 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:13:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[37680]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-74325-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[37680]: osdmap e148: 8 total, 8 up, 8 in 2024-04-27T01:13:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[37680]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[26714]: pgmap v179: 404 pgs: 112 unknown, 292 active+clean; 592 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:13:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[26714]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi162-74325-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[26714]: osdmap e148: 8 total, 8 up, 8 in 2024-04-27T01:13:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:56 smithi162 ceph-mon[26714]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:13:57.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:57 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:57.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:57 smithi181 ceph-mon[30237]: osdmap e149: 8 total, 8 up, 8 in 2024-04-27T01:13:57.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1875658228' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-74307-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:57.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:57 smithi181 ceph-mon[30237]: from='client.48859 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-74307-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:57.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:57.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:57 smithi181 ceph-mon[30237]: from='client.48859 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-74307-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:57.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:57 smithi181 ceph-mon[30237]: osdmap e150: 8 total, 8 up, 8 in 2024-04-27T01:13:57.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:57.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[37680]: osdmap e149: 8 total, 8 up, 8 in 2024-04-27T01:13:57.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1875658228' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-74307-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:57.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[37680]: from='client.48859 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-74307-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:57.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:57.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[37680]: from='client.48859 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-74307-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:57.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[37680]: osdmap e150: 8 total, 8 up, 8 in 2024-04-27T01:13:57.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74610-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:57.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[26714]: osdmap e149: 8 total, 8 up, 8 in 2024-04-27T01:13:57.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1875658228' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-74307-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:57.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[26714]: from='client.48859 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-74307-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:57.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:57.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[26714]: from='client.48859 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi162-74307-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:57.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:57 smithi162 ceph-mon[26714]: osdmap e150: 8 total, 8 up, 8 in 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP ( api_snapshots_pp: Running main() from gmock_main.cc 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: seed 74682 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (3514 ms) 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (3294 ms) 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (3525 ms) 2024-04-27T01:13:58.166 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1815 ms) 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (6194 ms) 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (18342 ms total) 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4882 ms) 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4125 ms) 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6610 ms) 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-04-27T01:13:58.167 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (5007 ms) 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2067 ms) 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2447-gb22e2ebd/rpm/el9/BUILD/ceph-19.0.0-2447-gb22e2ebd/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi162-74682-7 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: waiting for snaps to purge 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17930 ms) 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (40622 ms total) 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-04-27T01:13:58.168 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-04-27T01:13:58.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5068 ms) 2024-04-27T01:13:58.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5077 ms total) 2024-04-27T01:13:58.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-04-27T01:13:58.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-04-27T01:13:58.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-04-27T01:13:58.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2999 ms) 2024-04-27T01:13:58.169 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-04-27T01:13:58.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:58 smithi181 ceph-mon[30237]: pgmap v182: 372 pgs: 20 creating+peering, 32 unknown, 320 active+clean; 592 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 MiB/s wr, 2 op/s 2024-04-27T01:13:58.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2641890001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-74325-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:58.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:58.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2641890001' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-74325-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:58.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:58 smithi181 ceph-mon[30237]: osdmap e151: 8 total, 8 up, 8 in 2024-04-27T01:13:58.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:58.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:58 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:58.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[37680]: pgmap v182: 372 pgs: 20 creating+peering, 32 unknown, 320 active+clean; 592 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 MiB/s wr, 2 op/s 2024-04-27T01:13:58.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2641890001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-74325-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:58.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:58.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2641890001' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-74325-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:58.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[37680]: osdmap e151: 8 total, 8 up, 8 in 2024-04-27T01:13:58.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:58.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:58.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[26714]: pgmap v182: 372 pgs: 20 creating+peering, 32 unknown, 320 active+clean; 592 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 MiB/s wr, 2 op/s 2024-04-27T01:13:58.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2641890001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-74325-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:58.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:58.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2641890001' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi162-74325-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:13:58.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[26714]: osdmap e151: 8 total, 8 up, 8 in 2024-04-27T01:13:58.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:58.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:58 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:59.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:59.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]': finished 2024-04-27T01:13:59.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[37680]: osdmap e152: 8 total, 8 up, 8 in 2024-04-27T01:13:59.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:59.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:59.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2198844426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-74307-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:59.730 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[37680]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-74307-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:59.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:59.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]': finished 2024-04-27T01:13:59.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[26714]: osdmap e152: 8 total, 8 up, 8 in 2024-04-27T01:13:59.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:59.731 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:59.731 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2198844426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-74307-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:59.731 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-mon[26714]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-74307-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:59.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:13:59.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:59 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74610-24"}]': finished 2024-04-27T01:13:59.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:59 smithi181 ceph-mon[30237]: osdmap e152: 8 total, 8 up, 8 in 2024-04-27T01:13:59.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/385229752' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:59.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:59 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]: dispatch 2024-04-27T01:13:59.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2198844426' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-74307-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:13:59.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:13:59 smithi181 ceph-mon[30237]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-74307-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:00.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:13:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:13:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:14:00.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:00 smithi181 ceph-mon[30237]: pgmap v186: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 512 B/s wr, 1 op/s 2024-04-27T01:14:00.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:00.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:00 smithi181 ceph-mon[30237]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]': finished 2024-04-27T01:14:00.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:00 smithi181 ceph-mon[30237]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-74307-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:00.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:00 smithi181 ceph-mon[30237]: osdmap e153: 8 total, 8 up, 8 in 2024-04-27T01:14:00.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/361589954' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-74325-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:00.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:00 smithi181 ceph-mon[30237]: from='client.56621 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-74325-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[37680]: pgmap v186: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 512 B/s wr, 1 op/s 2024-04-27T01:14:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[37680]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]': finished 2024-04-27T01:14:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[37680]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-74307-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[37680]: osdmap e153: 8 total, 8 up, 8 in 2024-04-27T01:14:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/361589954' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-74325-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[37680]: from='client.56621 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-74325-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[26714]: pgmap v186: 332 pgs: 32 unknown, 300 active+clean; 592 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 512 B/s wr, 1 op/s 2024-04-27T01:14:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[26714]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74610-24"}]': finished 2024-04-27T01:14:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[26714]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi162-74307-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[26714]: osdmap e153: 8 total, 8 up, 8 in 2024-04-27T01:14:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/361589954' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-74325-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:00.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:00 smithi162 ceph-mon[26714]: from='client.56621 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-74325-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:01.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:01.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:01 smithi181 ceph-mon[30237]: from='client.56621 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-74325-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:01.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:01 smithi181 ceph-mon[30237]: osdmap e154: 8 total, 8 up, 8 in 2024-04-27T01:14:01.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/164624725' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-74610-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:01.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:01 smithi181 ceph-mon[30237]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-74610-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:01.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:01.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:01 smithi162 ceph-mon[37680]: from='client.56621 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-74325-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:01.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:01 smithi162 ceph-mon[37680]: osdmap e154: 8 total, 8 up, 8 in 2024-04-27T01:14:01.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/164624725' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-74610-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:01.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:01 smithi162 ceph-mon[37680]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-74610-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:01.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:01.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:01 smithi162 ceph-mon[26714]: from='client.56621 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi162-74325-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:01.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:01 smithi162 ceph-mon[26714]: osdmap e154: 8 total, 8 up, 8 in 2024-04-27T01:14:01.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/164624725' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-74610-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:01.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:01 smithi162 ceph-mon[26714]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-74610-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:02.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:14:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:14:02.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:02 smithi181 ceph-mon[30237]: pgmap v189: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 256 B/s wr, 0 op/s 2024-04-27T01:14:02.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:02.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:02 smithi181 ceph-mon[30237]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-74610-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:02.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:02 smithi181 ceph-mon[30237]: osdmap e155: 8 total, 8 up, 8 in 2024-04-27T01:14:02.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2626232440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-74307-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:02.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:02 smithi181 ceph-mon[30237]: from='client.48886 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-74307-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:02.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[26714]: pgmap v189: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 256 B/s wr, 0 op/s 2024-04-27T01:14:02.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:02.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[26714]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-74610-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:02.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[26714]: osdmap e155: 8 total, 8 up, 8 in 2024-04-27T01:14:02.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2626232440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-74307-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:02.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[26714]: from='client.48886 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-74307-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:02.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[37680]: pgmap v189: 364 pgs: 64 unknown, 300 active+clean; 592 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 256 B/s wr, 0 op/s 2024-04-27T01:14:02.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:02.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[37680]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi162-74610-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:02.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[37680]: osdmap e155: 8 total, 8 up, 8 in 2024-04-27T01:14:02.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2626232440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-74307-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:02.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:02 smithi162 ceph-mon[37680]: from='client.48886 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-74307-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:03 smithi181 ceph-mon[30237]: from='client.48886 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-74307-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:03 smithi181 ceph-mon[30237]: osdmap e156: 8 total, 8 up, 8 in 2024-04-27T01:14:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:03 smithi181 ceph-mon[30237]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1741758209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74325-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:03.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:03 smithi181 ceph-mon[30237]: from='client.56639 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74325-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[26714]: from='client.48886 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-74307-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[26714]: osdmap e156: 8 total, 8 up, 8 in 2024-04-27T01:14:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[26714]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1741758209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74325-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[26714]: from='client.56639 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74325-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:03.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:03.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:03.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[37680]: from='client.48886 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi162-74307-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:03.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[37680]: osdmap e156: 8 total, 8 up, 8 in 2024-04-27T01:14:03.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:03.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[37680]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:03.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1741758209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74325-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:03.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:03 smithi162 ceph-mon[37680]: from='client.56639 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74325-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:04 smithi181 ceph-mon[30237]: pgmap v192: 356 pgs: 14 creating+peering, 50 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:04 smithi181 ceph-mon[30237]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:04 smithi181 ceph-mon[30237]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]': finished 2024-04-27T01:14:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:04 smithi181 ceph-mon[30237]: from='client.56639 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74325-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:04 smithi181 ceph-mon[30237]: osdmap e157: 8 total, 8 up, 8 in 2024-04-27T01:14:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:04 smithi181 ceph-mon[30237]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3267891408' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-74610-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:04.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:04 smithi181 ceph-mon[30237]: from='client.56645 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-74610-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[26714]: pgmap v192: 356 pgs: 14 creating+peering, 50 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[26714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[26714]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]': finished 2024-04-27T01:14:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[26714]: from='client.56639 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74325-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[26714]: osdmap e157: 8 total, 8 up, 8 in 2024-04-27T01:14:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[26714]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:04.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3267891408' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-74610-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:04.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[26714]: from='client.56645 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-74610-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:04.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[37680]: pgmap v192: 356 pgs: 14 creating+peering, 50 unknown, 292 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:04.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:04.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[37680]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:04.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[37680]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi162-74682-16"}]': finished 2024-04-27T01:14:04.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[37680]: from='client.56639 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi162-74325-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:04.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[37680]: osdmap e157: 8 total, 8 up, 8 in 2024-04-27T01:14:04.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4235970137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:04.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[37680]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]: dispatch 2024-04-27T01:14:04.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3267891408' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-74610-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:04.841 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:04 smithi162 ceph-mon[37680]: from='client.56645 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-74610-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:05.139 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnaps snapshots: Running main() from gmock_main.cc 2024-04-27T01:14:05.139 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-04-27T01:14:05.139 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [----------] Global test environment set-up. 2024-04-27T01:14:05.139 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-04-27T01:14:05.139 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-04-27T01:14:05.140 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (13104 ms) 2024-04-27T01:14:05.140 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-04-27T01:14:05.140 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (6833 ms) 2024-04-27T01:14:05.140 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-04-27T01:14:05.140 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (5813 ms) 2024-04-27T01:14:05.141 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-04-27T01:14:05.141 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (6936 ms) 2024-04-27T01:14:05.141 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-04-27T01:14:05.141 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7701 ms) 2024-04-27T01:14:05.141 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (40387 ms total) 2024-04-27T01:14:05.141 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: 2024-04-27T01:14:05.142 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-04-27T01:14:05.142 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-04-27T01:14:05.142 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5600 ms) 2024-04-27T01:14:05.142 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-04-27T01:14:05.142 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (8021 ms) 2024-04-27T01:14:05.142 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-04-27T01:14:05.143 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8013 ms) 2024-04-27T01:14:05.143 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-04-27T01:14:05.143 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (5992 ms) 2024-04-27T01:14:05.143 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-04-27T01:14:05.143 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4011 ms) 2024-04-27T01:14:05.143 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-04-27T01:14:05.144 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi162-75629-11. 2024-04-27T01:14:05.144 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: Waiting for snaps to purge. 2024-04-27T01:14:05.144 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (20019 ms) 2024-04-27T01:14:05.144 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (51656 ms total) 2024-04-27T01:14:05.144 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: 2024-04-27T01:14:05.144 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [----------] Global test environment tear-down 2024-04-27T01:14:05.144 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (92043 ms total) 2024-04-27T01:14:05.145 INFO:tasks.workunit.client.0.smithi162.stdout: snapshots: [ PASSED ] 11 tests. 2024-04-27T01:14:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]': finished 2024-04-27T01:14:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.56645 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-74610-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: osdmap e158: 8 total, 8 up, 8 in 2024-04-27T01:14:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3120862834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-74307-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.48904 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-74307-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:05.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:05.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:05 smithi181 ceph-mon[30237]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]': finished 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.56645 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-74610-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: osdmap e158: 8 total, 8 up, 8 in 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3120862834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-74307-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.48904 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-74307-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[37680]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi162-74682-16"}]': finished 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.56645 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi162-74610-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: osdmap e158: 8 total, 8 up, 8 in 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3120862834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-74307-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.48904 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-74307-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:05.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:05.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:05 smithi162 ceph-mon[26714]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:06.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:06 smithi181 ceph-mon[30237]: pgmap v195: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:06.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:06.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:06 smithi181 ceph-mon[30237]: from='client.48904 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-74307-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:06.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:06 smithi181 ceph-mon[30237]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:06.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:06.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:06 smithi181 ceph-mon[30237]: osdmap e159: 8 total, 8 up, 8 in 2024-04-27T01:14:06.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1482188121' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-74325-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:06.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:06 smithi181 ceph-mon[30237]: from='client.56660 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-74325-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:06.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:06 smithi181 ceph-mon[30237]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[37680]: pgmap v195: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[37680]: from='client.48904 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-74307-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[37680]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[37680]: osdmap e159: 8 total, 8 up, 8 in 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1482188121' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-74325-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[37680]: from='client.56660 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-74325-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[37680]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[26714]: pgmap v195: 324 pgs: 64 unknown, 260 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[26714]: from='client.48904 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi162-74307-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[26714]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[26714]: osdmap e159: 8 total, 8 up, 8 in 2024-04-27T01:14:06.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1482188121' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-74325-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:06.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[26714]: from='client.56660 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-74325-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:06.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:06 smithi162 ceph-mon[26714]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:08.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:08.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:07 smithi181 ceph-mon[30237]: from='client.56660 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-74325-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:08.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:07 smithi181 ceph-mon[30237]: osdmap e160: 8 total, 8 up, 8 in 2024-04-27T01:14:08.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2106295532' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-74610-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:08.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:08.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:07 smithi162 ceph-mon[26714]: from='client.56660 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-74325-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:08.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:07 smithi162 ceph-mon[26714]: osdmap e160: 8 total, 8 up, 8 in 2024-04-27T01:14:08.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2106295532' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-74610-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:08.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:08.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:07 smithi162 ceph-mon[37680]: from='client.56660 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi162-74325-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:08.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:07 smithi162 ceph-mon[37680]: osdmap e160: 8 total, 8 up, 8 in 2024-04-27T01:14:08.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2106295532' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-74610-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:09.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:08 smithi181 ceph-mon[30237]: pgmap v198: 324 pgs: 19 creating+peering, 45 unknown, 260 active+clean; 592 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:09.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:09.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:08 smithi181 ceph-mon[30237]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]': finished 2024-04-27T01:14:09.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2106295532' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-74610-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:09.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:08 smithi181 ceph-mon[30237]: osdmap e161: 8 total, 8 up, 8 in 2024-04-27T01:14:09.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2399470544' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-74307-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[37680]: pgmap v198: 324 pgs: 19 creating+peering, 45 unknown, 260 active+clean; 592 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[37680]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]': finished 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2106295532' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-74610-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[37680]: osdmap e161: 8 total, 8 up, 8 in 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2399470544' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-74307-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[26714]: pgmap v198: 324 pgs: 19 creating+peering, 45 unknown, 260 active+clean; 592 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[26714]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]': finished 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2106295532' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi162-74610-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:09.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[26714]: osdmap e161: 8 total, 8 up, 8 in 2024-04-27T01:14:09.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2399470544' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-74307-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:10.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:10.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:10.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2399470544' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-74307-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:10.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: osdmap e162: 8 total, 8 up, 8 in 2024-04-27T01:14:10.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2301364862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-74325-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:10.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: from='client.56675 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-74325-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:10.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:10.012 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:09 smithi181 ceph-mon[30237]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:10.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:14:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:14:10.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:10.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:10.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2399470544' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-74307-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:10.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: osdmap e162: 8 total, 8 up, 8 in 2024-04-27T01:14:10.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2301364862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-74325-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: from='client.56675 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-74325-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[37680]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2399470544' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi162-74307-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:10.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: osdmap e162: 8 total, 8 up, 8 in 2024-04-27T01:14:10.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2301364862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-74325-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:10.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: from='client.56675 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-74325-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:10.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:10.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:09 smithi162 ceph-mon[26714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:10.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: pgmap v201: 332 pgs: 72 unknown, 260 active+clean; 592 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:10.989 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:10.990 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:14:10.990 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.56675 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-74325-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:10.990 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:10.990 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: osdmap e163: 8 total, 8 up, 8 in 2024-04-27T01:14:10.990 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-74610-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.990 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-74610-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:10.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:10.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:10.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:10.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-74307-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:10.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-74307-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:10.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:14:10.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:14:10.991 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: pgmap v201: 332 pgs: 72 unknown, 260 active+clean; 592 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.56675 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-74325-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: osdmap e163: 8 total, 8 up, 8 in 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-74610-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-74610-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-74307-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:10.992 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-74307-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:10.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:14:10.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:14:10.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:10 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:11.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: pgmap v201: 332 pgs: 72 unknown, 260 active+clean; 592 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:11.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:11.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:14:11.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.56675 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi162-74325-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:11.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:11.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: osdmap e163: 8 total, 8 up, 8 in 2024-04-27T01:14:11.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-74610-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:11.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-74610-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:11.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:11.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:11.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:11.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:11.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-74307-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:11.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-74307-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:11.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:14:11.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:14:11.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:10 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:11.643 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:11.644 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:11 smithi181 ceph-mon[30237]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-74307-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:11.644 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-74307-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:11.644 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:11 smithi181 ceph-mon[30237]: osdmap e164: 8 total, 8 up, 8 in 2024-04-27T01:14:12.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:11 smithi181 ceph-mon[30237]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-74307-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:12.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:12.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:11 smithi162 ceph-mon[37680]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-74307-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:12.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-74307-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:12.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:11 smithi162 ceph-mon[37680]: osdmap e164: 8 total, 8 up, 8 in 2024-04-27T01:14:12.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:11 smithi162 ceph-mon[37680]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-74307-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:12.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:12.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:11 smithi162 ceph-mon[26714]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi162-74307-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:12.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-74307-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:12.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:11 smithi162 ceph-mon[26714]: osdmap e164: 8 total, 8 up, 8 in 2024-04-27T01:14:12.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:11 smithi162 ceph-mon[26714]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-74307-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:12.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:14:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:14:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:12 smithi181 ceph-mon[30237]: pgmap v204: 268 pgs: 8 unknown, 260 active+clean; 592 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:12 smithi181 ceph-mon[30237]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-74610-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-74610-36"}]': finished 2024-04-27T01:14:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:12 smithi181 ceph-mon[30237]: osdmap e165: 8 total, 8 up, 8 in 2024-04-27T01:14:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1425243559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-74325-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:12 smithi181 ceph-mon[30237]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-74325-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[37680]: pgmap v204: 268 pgs: 8 unknown, 260 active+clean; 592 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[37680]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-74610-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-74610-36"}]': finished 2024-04-27T01:14:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[37680]: osdmap e165: 8 total, 8 up, 8 in 2024-04-27T01:14:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1425243559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-74325-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[37680]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-74325-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:13.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[26714]: pgmap v204: 268 pgs: 8 unknown, 260 active+clean; 592 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:13.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:13.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[26714]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi162-74610-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi162-74610-36"}]': finished 2024-04-27T01:14:13.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[26714]: osdmap e165: 8 total, 8 up, 8 in 2024-04-27T01:14:13.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1425243559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-74325-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:13.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:12 smithi162 ceph-mon[26714]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-74325-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:14.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:13 smithi181 ceph-mon[30237]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-74307-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-74307-27"}]': finished 2024-04-27T01:14:14.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:13 smithi181 ceph-mon[30237]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-74325-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:14.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:13 smithi181 ceph-mon[30237]: osdmap e166: 8 total, 8 up, 8 in 2024-04-27T01:14:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:13 smithi162 ceph-mon[37680]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-74307-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-74307-27"}]': finished 2024-04-27T01:14:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:13 smithi162 ceph-mon[37680]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-74325-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:14.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:13 smithi162 ceph-mon[37680]: osdmap e166: 8 total, 8 up, 8 in 2024-04-27T01:14:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:13 smithi162 ceph-mon[26714]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi162-74307-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi162-74307-27"}]': finished 2024-04-27T01:14:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:13 smithi162 ceph-mon[26714]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi162-74325-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:14.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:13 smithi162 ceph-mon[26714]: osdmap e166: 8 total, 8 up, 8 in 2024-04-27T01:14:15.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:14 smithi181 ceph-mon[30237]: pgmap v206: 308 pgs: 4 creating+peering, 36 unknown, 268 active+clean; 608 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:14:15.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:15.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:14 smithi181 ceph-mon[30237]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:15.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:14 smithi181 ceph-mon[30237]: osdmap e167: 8 total, 8 up, 8 in 2024-04-27T01:14:15.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:15.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:14 smithi181 ceph-mon[30237]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:15.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[37680]: pgmap v206: 308 pgs: 4 creating+peering, 36 unknown, 268 active+clean; 608 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:14:15.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:15.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[37680]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:15.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[37680]: osdmap e167: 8 total, 8 up, 8 in 2024-04-27T01:14:15.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:15.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[37680]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:15.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[26714]: pgmap v206: 308 pgs: 4 creating+peering, 36 unknown, 268 active+clean; 608 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:14:15.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:15.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[26714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:15.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[26714]: osdmap e167: 8 total, 8 up, 8 in 2024-04-27T01:14:15.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:15.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:14 smithi162 ceph-mon[26714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:16.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:16.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:15 smithi181 ceph-mon[30237]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]': finished 2024-04-27T01:14:16.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:16.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:15 smithi181 ceph-mon[30237]: osdmap e168: 8 total, 8 up, 8 in 2024-04-27T01:14:16.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2206236562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-74325-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:16.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:16.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:15 smithi181 ceph-mon[30237]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:16.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:15 smithi181 ceph-mon[30237]: from='client.56699 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-74325-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:16.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:15 smithi181 ceph-mon[30237]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:16.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:16.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[37680]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]': finished 2024-04-27T01:14:16.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:16.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[37680]: osdmap e168: 8 total, 8 up, 8 in 2024-04-27T01:14:16.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2206236562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-74325-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:16.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:16.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[37680]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:16.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[37680]: from='client.56699 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-74325-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:16.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[37680]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:16.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:16.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[26714]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi162-74610-36"}]': finished 2024-04-27T01:14:16.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3369024739' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:16.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[26714]: osdmap e168: 8 total, 8 up, 8 in 2024-04-27T01:14:16.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2206236562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-74325-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:16.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:16.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[26714]: from='client.48937 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]: dispatch 2024-04-27T01:14:16.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[26714]: from='client.56699 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-74325-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:16.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:15 smithi162 ceph-mon[26714]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:16.180 INFO:tasks.workunit.client.0.smithi162.stdout:67483 ms total) 2024-04-27T01:14:16.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-04-27T01:14:16.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-04-27T01:14:16.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-04-27T01:14:16.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (100 ms) 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (100 ms total) 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (33 ms) 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (5 ms) 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (38 ms total) 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (32 ms) 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (13 ms) 2024-04-27T01:14:16.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (45 ms total) 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (30 ms) 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (3 ms) 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (33 ms total) 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1041 ms) 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1041 ms total) 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: 2024-04-27T01:14:16.182 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-04-27T01:14:16.183 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (103394 ms total) 2024-04-27T01:14:16.183 INFO:tasks.workunit.client.0.smithi162.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-04-27T01:14:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:16 smithi181 ceph-mon[30237]: pgmap v209: 276 pgs: 8 unknown, 268 active+clean; 608 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:14:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:16 smithi181 ceph-mon[30237]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]': finished 2024-04-27T01:14:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:16 smithi181 ceph-mon[30237]: from='client.56699 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-74325-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:16 smithi181 ceph-mon[30237]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]': finished 2024-04-27T01:14:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:16 smithi181 ceph-mon[30237]: osdmap e169: 8 total, 8 up, 8 in 2024-04-27T01:14:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:16 smithi181 ceph-mon[30237]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:17.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:16 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:17.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[37680]: pgmap v209: 276 pgs: 8 unknown, 268 active+clean; 608 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:14:17.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:17.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[37680]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]': finished 2024-04-27T01:14:17.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[37680]: from='client.56699 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-74325-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:17.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[37680]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]': finished 2024-04-27T01:14:17.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[37680]: osdmap e169: 8 total, 8 up, 8 in 2024-04-27T01:14:17.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:17.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[37680]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[26714]: pgmap v209: 276 pgs: 8 unknown, 268 active+clean; 608 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[26714]: from='client.48937 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi162-74610-36"}]': finished 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[26714]: from='client.56699 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi162-74325-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[26714]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi162-74307-27"}]': finished 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[26714]: osdmap e169: 8 total, 8 up, 8 in 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3005931662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[26714]: from='client.48943 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]: dispatch 2024-04-27T01:14:17.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:17.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:16 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:17.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: Running main() from gmock_main.cc 2024-04-27T01:14:17.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-04-27T01:14:17.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] Global test environment set-up. 2024-04-27T01:14:17.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-04-27T01:14:17.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-04-27T01:14:17.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.TooBig (10545 ms) 2024-04-27T01:14:17.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-04-27T01:14:17.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (5267 ms) 2024-04-27T01:14:17.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-04-27T01:14:17.176 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (4558 ms) 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (5812 ms) 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3894 ms) 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (4256 ms) 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3267 ms) 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3798 ms) 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4423 ms) 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2854 ms) 2024-04-27T01:14:17.177 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-04-27T01:14:17.178 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3766 ms) 2024-04-27T01:14:17.178 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-04-27T01:14:17.178 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3003 ms) 2024-04-27T01:14:17.178 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-04-27T01:14:17.178 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2988 ms) 2024-04-27T01:14:17.178 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-04-27T01:14:17.178 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3022 ms) 2024-04-27T01:14:17.178 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-04-27T01:14:17.178 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.Flush (3254 ms) 2024-04-27T01:14:17.179 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-04-27T01:14:17.179 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2731 ms) 2024-04-27T01:14:17.179 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-04-27T01:14:17.179 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3009 ms) 2024-04-27T01:14:17.179 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-04-27T01:14:17.179 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3014 ms) 2024-04-27T01:14:17.179 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-04-27T01:14:17.179 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3018 ms) 2024-04-27T01:14:17.179 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3504 ms) 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (2515 ms) 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3006 ms) 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2942 ms) 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3015 ms) 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3259 ms) 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-04-27T01:14:17.180 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2770 ms) 2024-04-27T01:14:17.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] 26 tests from LibRadosAio (97491 ms total) 2024-04-27T01:14:17.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: 2024-04-27T01:14:17.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-04-27T01:14:17.181 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-04-27T01:14:18.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:18.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]': finished 2024-04-27T01:14:18.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:18.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: osdmap e170: 8 total, 8 up, 8 in 2024-04-27T01:14:18.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:14:18.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:14:18.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-74307-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:18.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:17 smithi181 ceph-mon[30237]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-74307-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:18.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:18.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]': finished 2024-04-27T01:14:18.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:18.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: osdmap e170: 8 total, 8 up, 8 in 2024-04-27T01:14:18.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:14:18.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:14:18.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-74307-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:18.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[37680]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-74307-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:18.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:18.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.48943 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi162-74307-27"}]': finished 2024-04-27T01:14:18.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:18.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: osdmap e170: 8 total, 8 up, 8 in 2024-04-27T01:14:18.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:14:18.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13"}]: dispatch 2024-04-27T01:14:18.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:18.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-74307-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:18.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:17 smithi162 ceph-mon[26714]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-74307-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:18 smithi181 ceph-mon[30237]: pgmap v212: 300 pgs: 14 unknown, 286 active+clean; 588 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-27T01:14:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:18 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:18 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13"}]': finished 2024-04-27T01:14:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:18 smithi181 ceph-mon[30237]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-74307-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:18 smithi181 ceph-mon[30237]: osdmap e171: 8 total, 8 up, 8 in 2024-04-27T01:14:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-74307-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/116506158' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-74325-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:18 smithi181 ceph-mon[30237]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-74307-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:19.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:18 smithi181 ceph-mon[30237]: from='client.48964 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-74325-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:19.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[37680]: pgmap v212: 300 pgs: 14 unknown, 286 active+clean; 588 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-27T01:14:19.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:19.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:19.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13"}]': finished 2024-04-27T01:14:19.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[37680]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-74307-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:19.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[37680]: osdmap e171: 8 total, 8 up, 8 in 2024-04-27T01:14:19.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-74307-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:19.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/116506158' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-74325-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:19.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[37680]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-74307-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:19.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[37680]: from='client.48964 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-74325-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:19.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[26714]: pgmap v212: 300 pgs: 14 unknown, 286 active+clean; 588 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-27T01:14:19.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:19.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:19.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-13"}]': finished 2024-04-27T01:14:19.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[26714]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi162-74307-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:19.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[26714]: osdmap e171: 8 total, 8 up, 8 in 2024-04-27T01:14:19.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-74307-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:19.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/116506158' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-74325-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:19.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[26714]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-74307-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:19.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:18 smithi162 ceph-mon[26714]: from='client.48964 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-74325-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:20.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:20.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:19 smithi181 ceph-mon[30237]: from='client.48964 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-74325-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:20.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:19 smithi181 ceph-mon[30237]: osdmap e172: 8 total, 8 up, 8 in 2024-04-27T01:14:20.088 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:14:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:14:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:14:20.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:20.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:19 smithi162 ceph-mon[26714]: from='client.48964 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-74325-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:20.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:19 smithi162 ceph-mon[26714]: osdmap e172: 8 total, 8 up, 8 in 2024-04-27T01:14:20.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:20.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:19 smithi162 ceph-mon[37680]: from='client.48964 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi162-74325-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:20.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:19 smithi162 ceph-mon[37680]: osdmap e172: 8 total, 8 up, 8 in 2024-04-27T01:14:21.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:20 smithi181 ceph-mon[30237]: pgmap v216: 268 pgs: 32 unknown, 236 active+clean; 610 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-04-27T01:14:21.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:21.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:20 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:21.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:21.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:20 smithi181 ceph-mon[30237]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-74307-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-74307-28"}]': finished 2024-04-27T01:14:21.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:20 smithi181 ceph-mon[30237]: osdmap e173: 8 total, 8 up, 8 in 2024-04-27T01:14:21.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:21.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:20 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[37680]: pgmap v216: 268 pgs: 32 unknown, 236 active+clean; 610 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[37680]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-74307-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-74307-28"}]': finished 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[37680]: osdmap e173: 8 total, 8 up, 8 in 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[26714]: pgmap v216: 268 pgs: 32 unknown, 236 active+clean; 610 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:21.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[26714]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi162-74307-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi162-74307-28"}]': finished 2024-04-27T01:14:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[26714]: osdmap e173: 8 total, 8 up, 8 in 2024-04-27T01:14:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:21.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:20 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:22.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:22.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:21 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:22.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:21 smithi181 ceph-mon[30237]: osdmap e174: 8 total, 8 up, 8 in 2024-04-27T01:14:22.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:22.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2671763481' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-74325-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:22.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:21 smithi181 ceph-mon[30237]: from='client.56717 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-74325-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:22.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:21 smithi181 ceph-mon[30237]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:22.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:22.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:22.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[37680]: osdmap e174: 8 total, 8 up, 8 in 2024-04-27T01:14:22.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:22.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2671763481' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-74325-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:22.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[37680]: from='client.56717 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-74325-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:22.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[37680]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:22.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:22.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:22.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[26714]: osdmap e174: 8 total, 8 up, 8 in 2024-04-27T01:14:22.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:22.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2671763481' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-74325-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:22.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[26714]: from='client.56717 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-74325-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:22.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:21 smithi162 ceph-mon[26714]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:22.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:14:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: pgmap v219: 300 pgs: 72 unknown, 228 active+clean; 586 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: from='client.56717 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-74325-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]': finished 2024-04-27T01:14:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: osdmap e175: 8 total, 8 up, 8 in 2024-04-27T01:14:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:23.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:23.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:23.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:22 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: pgmap v219: 300 pgs: 72 unknown, 228 active+clean; 586 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: from='client.56717 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-74325-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]': finished 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: osdmap e175: 8 total, 8 up, 8 in 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:23.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: pgmap v219: 300 pgs: 72 unknown, 228 active+clean; 586 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: from='client.56717 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi162-74325-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]': finished 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1356721458' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: osdmap e175: 8 total, 8 up, 8 in 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: from='client.48910 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]: dispatch 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:23.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:23.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:22 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:23.122 INFO:tasks.workunit.client.0.smithi162.stdout:hotsECPP.SnapRemovePP (2009 ms) 2024-04-27T01:14:23.122 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-04-27T01:14:23.122 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1931 ms) 2024-04-27T01:14:23.122 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2017 ms) 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8957 ms total) 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4248 ms) 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3777 ms) 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3928 ms) 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11953 ms total) 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (110342 ms total) 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-04-27T01:14:23.123 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-04-27T01:14:23.124 INFO:tasks.workunit.client.0.smithi162.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-27T01:14:24.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:24.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:24.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:23 smithi181 ceph-mon[30237]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]': finished 2024-04-27T01:14:24.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:23 smithi181 ceph-mon[30237]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]': finished 2024-04-27T01:14:24.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:23 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:24.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:24.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:24.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:23 smithi181 ceph-mon[30237]: osdmap e176: 8 total, 8 up, 8 in 2024-04-27T01:14:24.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:23 smithi181 ceph-mon[30237]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:24.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:23 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:24.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:24.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:24.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[37680]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]': finished 2024-04-27T01:14:24.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[37680]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]': finished 2024-04-27T01:14:24.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:24.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:24.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:24.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[37680]: osdmap e176: 8 total, 8 up, 8 in 2024-04-27T01:14:24.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[37680]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:24.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:24.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:24.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:24.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[26714]: from='client.48910 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi162-74682-21"}]': finished 2024-04-27T01:14:24.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[26714]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi162-74307-28"}]': finished 2024-04-27T01:14:24.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:24.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2603795457' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:24.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:24.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[26714]: osdmap e176: 8 total, 8 up, 8 in 2024-04-27T01:14:24.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[26714]: from='client.48961 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]: dispatch 2024-04-27T01:14:24.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:23 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:25.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: pgmap v222: 260 pgs: 260 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:25.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:25.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]': finished 2024-04-27T01:14:25.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-15"}]': finished 2024-04-27T01:14:25.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-15", "mode": "writeback"}]: dispatch 2024-04-27T01:14:25.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: osdmap e177: 8 total, 8 up, 8 in 2024-04-27T01:14:25.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3358821673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-74325-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:25.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-15", "mode": "writeback"}]: dispatch 2024-04-27T01:14:25.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-74325-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:25.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-74307-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:25.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:24 smithi181 ceph-mon[30237]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-74307-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:25.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: pgmap v222: 260 pgs: 260 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:25.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:25.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]': finished 2024-04-27T01:14:25.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-15"}]': finished 2024-04-27T01:14:25.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-15", "mode": "writeback"}]: dispatch 2024-04-27T01:14:25.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: osdmap e177: 8 total, 8 up, 8 in 2024-04-27T01:14:25.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3358821673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-74325-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:25.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-15", "mode": "writeback"}]: dispatch 2024-04-27T01:14:25.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-74325-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:25.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-74307-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:25.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[37680]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-74307-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:25.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: pgmap v222: 260 pgs: 260 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:25.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.48961 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi162-74307-28"}]': finished 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-15"}]': finished 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-15", "mode": "writeback"}]: dispatch 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: osdmap e177: 8 total, 8 up, 8 in 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3358821673' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-74325-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-15", "mode": "writeback"}]: dispatch 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.56723 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-74325-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:25.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-74307-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:25.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:24 smithi162 ceph-mon[26714]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-74307-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-15", "mode": "writeback"}]': finished 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-74325-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-74307-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]': finished 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: osdmap e178: 8 total, 8 up, 8 in 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-74307-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:26.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-74307-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:26.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:26.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:26.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:25 smithi181 ceph-mon[30237]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:26.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:26.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:26.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:26.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:26.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-15", "mode": "writeback"}]': finished 2024-04-27T01:14:26.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-74325-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:26.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-74307-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:26.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]': finished 2024-04-27T01:14:26.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: osdmap e178: 8 total, 8 up, 8 in 2024-04-27T01:14:26.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-74307-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-74307-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[26714]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-15", "mode": "writeback"}]': finished 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.56723 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi162-74325-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:26.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi162-74307-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:26.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]': finished 2024-04-27T01:14:26.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: osdmap e178: 8 total, 8 up, 8 in 2024-04-27T01:14:26.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-74307-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:26.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:26.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-74307-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:26.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:26.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:26.090 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:25 smithi162 ceph-mon[37680]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (8989 ms) 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (24135 ms) 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6634 ms) 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-04-27T01:14:26.350 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3855 ms) 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3804 ms) 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3011 ms) 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3010 ms) 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2987 ms) 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3008 ms) 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3020 ms) 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-04-27T01:14:26.351 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3017 ms) 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2967 ms) 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3011 ms) 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3024 ms) 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3006 ms) 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3014 ms) 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3005 ms) 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-04-27T01:14:26.352 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2932 ms) 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3023 ms) 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3033 ms) 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2998 ms) 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3243 ms) 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2777 ms) 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3010 ms) 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2925 ms) 2024-04-27T01:14:26.353 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-04-27T01:14:26.354 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3020 ms) 2024-04-27T01:14:26.354 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-04-27T01:14:27.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:26 smithi181 ceph-mon[30237]: pgmap v225: 292 pgs: 32 unknown, 260 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:27.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:26 smithi181 ceph-mon[30237]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]': finished 2024-04-27T01:14:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:26 smithi181 ceph-mon[30237]: osdmap e179: 8 total, 8 up, 8 in 2024-04-27T01:14:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:27.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:26 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:27.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[37680]: pgmap v225: 292 pgs: 32 unknown, 260 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:27.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[37680]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]': finished 2024-04-27T01:14:27.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:27.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[37680]: osdmap e179: 8 total, 8 up, 8 in 2024-04-27T01:14:27.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:27.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:27.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[26714]: pgmap v225: 292 pgs: 32 unknown, 260 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:27.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[26714]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]': finished 2024-04-27T01:14:27.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:27.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[26714]: osdmap e179: 8 total, 8 up, 8 in 2024-04-27T01:14:27.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:27.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:26 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:28.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:28 smithi181 ceph-mon[30237]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-74307-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-74307-29"}]': finished 2024-04-27T01:14:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:28 smithi181 ceph-mon[30237]: osdmap e180: 8 total, 8 up, 8 in 2024-04-27T01:14:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1899143482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-74325-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:28 smithi181 ceph-mon[30237]: pgmap v228: 300 pgs: 6 creating+peering, 34 unknown, 260 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:28 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[37680]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-74307-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-74307-29"}]': finished 2024-04-27T01:14:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[37680]: osdmap e180: 8 total, 8 up, 8 in 2024-04-27T01:14:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1899143482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-74325-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[37680]: pgmap v228: 300 pgs: 6 creating+peering, 34 unknown, 260 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[26714]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi162-74307-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi162-74307-29"}]': finished 2024-04-27T01:14:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[26714]: osdmap e180: 8 total, 8 up, 8 in 2024-04-27T01:14:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1899143482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-74325-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15"}]: dispatch 2024-04-27T01:14:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[26714]: pgmap v228: 300 pgs: 6 creating+peering, 34 unknown, 260 active+clean; 586 KiB data, 546 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:28 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:29.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:29 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:29.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:29.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1899143482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-74325-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:29.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:29 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15"}]': finished 2024-04-27T01:14:29.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:29 smithi181 ceph-mon[30237]: osdmap e181: 8 total, 8 up, 8 in 2024-04-27T01:14:29.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:29.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:29.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:29.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1899143482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-74325-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:29.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15"}]': finished 2024-04-27T01:14:29.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[37680]: osdmap e181: 8 total, 8 up, 8 in 2024-04-27T01:14:29.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:29.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:29.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:29.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1899143482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi162-74325-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:29.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-15"}]': finished 2024-04-27T01:14:29.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[26714]: osdmap e181: 8 total, 8 up, 8 in 2024-04-27T01:14:29.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:30.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:14:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:14:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:14:30.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:30 smithi181 ceph-mon[30237]: osdmap e182: 8 total, 8 up, 8 in 2024-04-27T01:14:30.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:30.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:30 smithi181 ceph-mon[30237]: pgmap v231: 228 pgs: 228 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:30.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:30 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:30.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:30 smithi181 ceph-mon[30237]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:30.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:30 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:30.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:30.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[37680]: osdmap e182: 8 total, 8 up, 8 in 2024-04-27T01:14:30.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:30.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[37680]: pgmap v231: 228 pgs: 228 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:30.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:30.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[37680]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:30.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:30.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:30.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[26714]: osdmap e182: 8 total, 8 up, 8 in 2024-04-27T01:14:30.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:30.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[26714]: pgmap v231: 228 pgs: 228 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:30.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:30.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[26714]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:30.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:30.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:31.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:31 smithi181 ceph-mon[30237]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]': finished 2024-04-27T01:14:31.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:31 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-04-27T01:14:31.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:31 smithi181 ceph-mon[30237]: osdmap e183: 8 total, 8 up, 8 in 2024-04-27T01:14:31.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:31.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2694294210' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-74325-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:31 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:31 smithi181 ceph-mon[30237]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:31.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:31 smithi181 ceph-mon[30237]: from='client.56741 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-74325-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:31.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[37680]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]': finished 2024-04-27T01:14:31.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-04-27T01:14:31.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[37680]: osdmap e183: 8 total, 8 up, 8 in 2024-04-27T01:14:31.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:31.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2694294210' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-74325-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[37680]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:31.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[37680]: from='client.56741 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-74325-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:31.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[26714]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi162-74307-29"}]': finished 2024-04-27T01:14:31.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-04-27T01:14:31.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[26714]: osdmap e183: 8 total, 8 up, 8 in 2024-04-27T01:14:31.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2151752698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:31.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2694294210' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-74325-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[26714]: from='client.56729 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]: dispatch 2024-04-27T01:14:31.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[26714]: from='client.56741 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-74325-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:31.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:32.410 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]': finished 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.56741 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-74325-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: osdmap e184: 8 total, 8 up, 8 in 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: pgmap v234: 292 pgs: 64 unknown, 228 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-74307-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:32.411 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-74307-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:32.412 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:32.412 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:32.412 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:32 smithi181 ceph-mon[30237]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:32.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]': finished 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.56741 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-74325-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: osdmap e184: 8 total, 8 up, 8 in 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: pgmap v234: 292 pgs: 64 unknown, 228 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-74307-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-74307-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[37680]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.56729 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi162-74307-29"}]': finished 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.56741 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi162-74325-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: osdmap e184: 8 total, 8 up, 8 in 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: pgmap v234: 292 pgs: 64 unknown, 228 active+clean; 586 KiB data, 547 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:32.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-74307-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-74307-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:32.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:32 smithi162 ceph-mon[26714]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]: dispatch 2024-04-27T01:14:33.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:33.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-74307-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:33.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]': finished 2024-04-27T01:14:33.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: osdmap e185: 8 total, 8 up, 8 in 2024-04-27T01:14:33.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:33.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-74307-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:33.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:33.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:33.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:33.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-74307-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:33.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:33.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:33.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:33.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-17"}]': finished 2024-04-27T01:14:33.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:33 smithi181 ceph-mon[30237]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]': finished 2024-04-27T01:14:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-74307-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]': finished 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: osdmap e185: 8 total, 8 up, 8 in 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-74307-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-74307-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:33.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-17"}]': finished 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[37680]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]': finished 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi162-74307-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pg_num","val":"11"}]': finished 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: osdmap e185: 8 total, 8 up, 8 in 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-74307-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:33.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:33.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-74307-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:33.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:33.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.34694 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-27T01:14:33.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:33.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-17"}]': finished 2024-04-27T01:14:33.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:33 smithi162 ceph-mon[26714]: from='client.34694 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi162-74405-1","var":"pgp_num","val":"11"}]': finished 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-17", "mode": "writeback"}]: dispatch 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: pgmap v237: 292 pgs: 32 unknown, 260 active+clean; 586 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: osdmap e186: 8 total, 8 up, 8 in 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-17", "mode": "writeback"}]: dispatch 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3577057482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-74325-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-74307-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-74307-30"}]': finished 2024-04-27T01:14:34.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-17", "mode": "writeback"}]': finished 2024-04-27T01:14:34.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3577057482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-74325-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:34.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-27T01:14:34.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: osdmap e187: 8 total, 8 up, 8 in 2024-04-27T01:14:34.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:34.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:34.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:34.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-17", "mode": "writeback"}]: dispatch 2024-04-27T01:14:34.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:34.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: pgmap v237: 292 pgs: 32 unknown, 260 active+clean; 586 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-04-27T01:14:34.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:34.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: osdmap e186: 8 total, 8 up, 8 in 2024-04-27T01:14:34.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-17", "mode": "writeback"}]: dispatch 2024-04-27T01:14:34.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3577057482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-74325-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-74307-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-74307-30"}]': finished 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-17", "mode": "writeback"}]': finished 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3577057482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-74325-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: osdmap e187: 8 total, 8 up, 8 in 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:34.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-17", "mode": "writeback"}]: dispatch 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: pgmap v237: 292 pgs: 32 unknown, 260 active+clean; 586 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: osdmap e186: 8 total, 8 up, 8 in 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-17", "mode": "writeback"}]: dispatch 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3577057482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-74325-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi162-74307-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi162-74307-30"}]': finished 2024-04-27T01:14:34.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-17", "mode": "writeback"}]': finished 2024-04-27T01:14:34.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3577057482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi162-74325-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:34.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-27T01:14:34.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: osdmap e187: 8 total, 8 up, 8 in 2024-04-27T01:14:34.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:34.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:34.841 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:36.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:35 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:36.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:35 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:36.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:35 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:36.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:35 smithi181 ceph-mon[30237]: osdmap e188: 8 total, 8 up, 8 in 2024-04-27T01:14:36.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:36.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:35 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:36.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:36.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:36.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:36.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:36.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[26714]: osdmap e188: 8 total, 8 up, 8 in 2024-04-27T01:14:36.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:36.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:36.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:36.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:36.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:36.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:36.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[37680]: osdmap e188: 8 total, 8 up, 8 in 2024-04-27T01:14:36.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:36.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17"}]: dispatch 2024-04-27T01:14:36.339 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:37.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:36 smithi181 ceph-mon[30237]: pgmap v240: 268 pgs: 8 unknown, 260 active+clean; 586 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-04-27T01:14:37.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:36 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:37.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:36 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17"}]': finished 2024-04-27T01:14:37.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:36 smithi181 ceph-mon[30237]: osdmap e189: 8 total, 8 up, 8 in 2024-04-27T01:14:37.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:37.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:36 smithi181 ceph-mon[30237]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:37.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi162-74325-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:37.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:37.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[37680]: pgmap v240: 268 pgs: 8 unknown, 260 active+clean; 586 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-04-27T01:14:37.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:37.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17"}]': finished 2024-04-27T01:14:37.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[37680]: osdmap e189: 8 total, 8 up, 8 in 2024-04-27T01:14:37.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:37.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[37680]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:37.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi162-74325-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:37.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:37.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[26714]: pgmap v240: 268 pgs: 8 unknown, 260 active+clean; 586 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-04-27T01:14:37.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:37.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-17"}]': finished 2024-04-27T01:14:37.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[26714]: osdmap e189: 8 total, 8 up, 8 in 2024-04-27T01:14:37.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:37.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[26714]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:37.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi162-74325-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:37.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]': finished 2024-04-27T01:14:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi162-74325-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: osdmap e190: 8 total, 8 up, 8 in 2024-04-27T01:14:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: pgmap v243: 259 pgs: 16 creating+peering, 1 peering, 9 unknown, 233 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd=[{ 2024-04-27T01:14:38.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: "prefix": "osd pool set", 2024-04-27T01:14:38.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: "pool": "PoolEIOFlag_smithi162-74325-33", 2024-04-27T01:14:38.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: "var": "eio", 2024-04-27T01:14:38.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: "val": "true" 2024-04-27T01:14:38.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:38 smithi181 ceph-mon[30237]: }]: dispatch 2024-04-27T01:14:38.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]': finished 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi162-74325-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: osdmap e190: 8 total, 8 up, 8 in 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: pgmap v243: 259 pgs: 16 creating+peering, 1 peering, 9 unknown, 233 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd=[{ 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: "prefix": "osd pool set", 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: "pool": "PoolEIOFlag_smithi162-74325-33", 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: "var": "eio", 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: "val": "true" 2024-04-27T01:14:38.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[37680]: }]: dispatch 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi162-74307-30"}]': finished 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi162-74325-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1393642429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: osdmap e190: 8 total, 8 up, 8 in 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: from='client.49000 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]: dispatch 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: pgmap v243: 259 pgs: 16 creating+peering, 1 peering, 9 unknown, 233 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd=[{ 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: "prefix": "osd pool set", 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: "pool": "PoolEIOFlag_smithi162-74325-33", 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: "var": "eio", 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: "val": "true" 2024-04-27T01:14:38.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:38 smithi162 ceph-mon[26714]: }]: dispatch 2024-04-27T01:14:39.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]': finished 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd='[{ 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: "prefix": "osd pool set", 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: "pool": "PoolEIOFlag_smithi162-74325-33", 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: "var": "eio", 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: "val": "true" 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: }]': finished 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: osdmap e191: 8 total, 8 up, 8 in 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-74307-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:39.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-74307-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:39.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:39.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]': finished 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd='[{ 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: "prefix": "osd pool set", 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: "pool": "PoolEIOFlag_smithi162-74325-33", 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: "var": "eio", 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: "val": "true" 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: }]': finished 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: osdmap e191: 8 total, 8 up, 8 in 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-74307-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-74307-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.49000 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi162-74307-30"}]': finished 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/108770324' entity='client.admin' cmd='[{ 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: "prefix": "osd pool set", 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: "pool": "PoolEIOFlag_smithi162-74325-33", 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: "var": "eio", 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: "val": "true" 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: }]': finished 2024-04-27T01:14:39.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: osdmap e191: 8 total, 8 up, 8 in 2024-04-27T01:14:39.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:39.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:39.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:39.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-74307-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:39.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-74307-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:39.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:40.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:14:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:14:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:14:40.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:40.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:40 smithi181 ceph-mon[30237]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-74307-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:40.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:40 smithi181 ceph-mon[30237]: osdmap e192: 8 total, 8 up, 8 in 2024-04-27T01:14:40.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-74307-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:40.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:40.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:40.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:40 smithi181 ceph-mon[30237]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-74307-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:40.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:40.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:40 smithi181 ceph-mon[30237]: pgmap v246: 259 pgs: 32 unknown, 1 peering, 226 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:40.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:40.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[26714]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-74307-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:40.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[26714]: osdmap e192: 8 total, 8 up, 8 in 2024-04-27T01:14:40.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-74307-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:40.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:40.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:40.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[26714]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-74307-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:40.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:40.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[26714]: pgmap v246: 259 pgs: 32 unknown, 1 peering, 226 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:40.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:40.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[37680]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi162-74307-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:40.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[37680]: osdmap e192: 8 total, 8 up, 8 in 2024-04-27T01:14:40.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-74307-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:40.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:40.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:40.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[37680]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-74307-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:40.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:40.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:40 smithi162 ceph-mon[37680]: pgmap v246: 259 pgs: 32 unknown, 1 peering, 226 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:41.490 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:41.490 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:41.490 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:41.490 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[37680]: osdmap e193: 8 total, 8 up, 8 in 2024-04-27T01:14:41.490 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:41.490 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/903497004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-74325-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:41.490 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:41.490 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:41.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:41.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[26714]: osdmap e193: 8 total, 8 up, 8 in 2024-04-27T01:14:41.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:41.491 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/903497004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-74325-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:41.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:41 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:41.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:41.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:41.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:41 smithi181 ceph-mon[30237]: osdmap e193: 8 total, 8 up, 8 in 2024-04-27T01:14:41.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:41 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:41.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/903497004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-74325-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:42.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:14:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:14:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:42 smithi181 ceph-mon[30237]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-74307-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-74307-31"}]': finished 2024-04-27T01:14:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:42 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-19"}]': finished 2024-04-27T01:14:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/903497004' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-74325-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:42.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:42 smithi181 ceph-mon[30237]: osdmap e194: 8 total, 8 up, 8 in 2024-04-27T01:14:42.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-19", "mode": "writeback"}]: dispatch 2024-04-27T01:14:42.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:42 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-19", "mode": "writeback"}]: dispatch 2024-04-27T01:14:42.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:42.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:42 smithi181 ceph-mon[30237]: pgmap v249: 299 pgs: 72 unknown, 1 peering, 226 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-04-27T01:14:42.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[37680]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-74307-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-74307-31"}]': finished 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-19"}]': finished 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/903497004' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-74325-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[37680]: osdmap e194: 8 total, 8 up, 8 in 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-19", "mode": "writeback"}]: dispatch 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-19", "mode": "writeback"}]: dispatch 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[37680]: pgmap v249: 299 pgs: 72 unknown, 1 peering, 226 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[26714]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi162-74307-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi162-74307-31"}]': finished 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-19"}]': finished 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/903497004' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi162-74325-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[26714]: osdmap e194: 8 total, 8 up, 8 in 2024-04-27T01:14:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-19", "mode": "writeback"}]: dispatch 2024-04-27T01:14:42.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-19", "mode": "writeback"}]: dispatch 2024-04-27T01:14:42.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:42.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:42 smithi162 ceph-mon[26714]: pgmap v249: 299 pgs: 72 unknown, 1 peering, 226 active+clean; 586 KiB data, 556 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-04-27T01:14:43.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:43 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:43.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:43 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:43.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:43 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-19", "mode": "writeback"}]': finished 2024-04-27T01:14:43.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:43 smithi181 ceph-mon[30237]: osdmap e195: 8 total, 8 up, 8 in 2024-04-27T01:14:43.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:43.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:43.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:43.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:43 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:43.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:43.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:43.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-19", "mode": "writeback"}]': finished 2024-04-27T01:14:43.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[37680]: osdmap e195: 8 total, 8 up, 8 in 2024-04-27T01:14:43.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:43.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:43.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:43.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:43.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:43.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:43.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-19", "mode": "writeback"}]': finished 2024-04-27T01:14:43.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[26714]: osdmap e195: 8 total, 8 up, 8 in 2024-04-27T01:14:43.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:43.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:43.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:43.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:43 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: pgmap v252: 291 pgs: 32 unknown, 259 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: osdmap e196: 8 total, 8 up, 8 in 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3352908307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-74325-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.49030 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-74325-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:44.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]': finished 2024-04-27T01:14:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19"}]': finished 2024-04-27T01:14:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.49030 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-74325-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: osdmap e197: 8 total, 8 up, 8 in 2024-04-27T01:14:44.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: pgmap v252: 291 pgs: 32 unknown, 259 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: osdmap e196: 8 total, 8 up, 8 in 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3352908307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-74325-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.49030 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-74325-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:44.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]': finished 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19"}]': finished 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.49030 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-74325-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: osdmap e197: 8 total, 8 up, 8 in 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: pgmap v252: 291 pgs: 32 unknown, 259 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: osdmap e196: 8 total, 8 up, 8 in 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:44.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:44.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3352908307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-74325-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:44.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19"}]: dispatch 2024-04-27T01:14:44.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.49030 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-74325-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:44.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:44.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi162-74307-31"}]': finished 2024-04-27T01:14:44.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-19"}]': finished 2024-04-27T01:14:44.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.49030 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi162-74325-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:44.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: osdmap e197: 8 total, 8 up, 8 in 2024-04-27T01:14:44.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:45.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:45.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:45 smithi181 ceph-mon[30237]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:45.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:45.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:45.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:45 smithi162 ceph-mon[37680]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:45.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:45.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2558399260' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:45.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:45 smithi162 ceph-mon[26714]: from='client.56762 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]: dispatch 2024-04-27T01:14:45.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:46.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:46 smithi181 ceph-mon[30237]: pgmap v254: 291 pgs: 32 unknown, 259 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:46 smithi181 ceph-mon[30237]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]': finished 2024-04-27T01:14:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:46 smithi181 ceph-mon[30237]: osdmap e198: 8 total, 8 up, 8 in 2024-04-27T01:14:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:46 smithi181 ceph-mon[30237]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:46 smithi181 ceph-mon[30237]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-74307-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:46 smithi181 ceph-mon[30237]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-74307-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:46.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[37680]: pgmap v254: 291 pgs: 32 unknown, 259 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[37680]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]': finished 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[37680]: osdmap e198: 8 total, 8 up, 8 in 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[37680]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[37680]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-74307-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[37680]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-74307-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[26714]: pgmap v254: 291 pgs: 32 unknown, 259 active+clean; 586 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:46.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[26714]: from='client.56762 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi162-74307-31"}]': finished 2024-04-27T01:14:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[26714]: osdmap e198: 8 total, 8 up, 8 in 2024-04-27T01:14:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[26714]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[26714]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-74307-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[26714]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-74307-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:46.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:47.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:47 smithi181 ceph-mon[30237]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-74307-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:47.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-74307-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:47.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:47 smithi181 ceph-mon[30237]: osdmap e199: 8 total, 8 up, 8 in 2024-04-27T01:14:47.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:47 smithi181 ceph-mon[30237]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-74307-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:47.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:47 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2578485027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:47 smithi181 ceph-mon[30237]: from='client.56786 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:47.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[37680]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-74307-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:47.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-74307-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:47.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[37680]: osdmap e199: 8 total, 8 up, 8 in 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[37680]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-74307-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2578485027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[37680]: from='client.56786 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[26714]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi162-74307-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-74307-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[26714]: osdmap e199: 8 total, 8 up, 8 in 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[26714]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-74307-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2578485027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[26714]: from='client.56786 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:47.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:48.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:48 smithi181 ceph-mon[30237]: pgmap v257: 291 pgs: 44 creating+peering, 20 unknown, 227 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:48.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:48 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:48.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:48 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:48.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:48 smithi181 ceph-mon[30237]: from='client.56786 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:48.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:48.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:48 smithi181 ceph-mon[30237]: osdmap e200: 8 total, 8 up, 8 in 2024-04-27T01:14:48.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:48 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:48.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[37680]: pgmap v257: 291 pgs: 44 creating+peering, 20 unknown, 227 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:48.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:48.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:48.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[37680]: from='client.56786 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:48.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:48.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[37680]: osdmap e200: 8 total, 8 up, 8 in 2024-04-27T01:14:48.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:48.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[26714]: pgmap v257: 291 pgs: 44 creating+peering, 20 unknown, 227 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:14:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:48.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[26714]: from='client.56786 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:48.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:48.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[26714]: osdmap e200: 8 total, 8 up, 8 in 2024-04-27T01:14:48.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:48.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:49.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:49 smithi181 ceph-mon[30237]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-74307-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-74307-32"}]': finished 2024-04-27T01:14:49.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:49 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:49 smithi181 ceph-mon[30237]: osdmap e201: 8 total, 8 up, 8 in 2024-04-27T01:14:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:49 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/134288650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:49 smithi181 ceph-mon[30237]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:49.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[37680]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-74307-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-74307-32"}]': finished 2024-04-27T01:14:49.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[37680]: osdmap e201: 8 total, 8 up, 8 in 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/134288650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[37680]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[26714]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi162-74307-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi162-74307-32"}]': finished 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[26714]: osdmap e201: 8 total, 8 up, 8 in 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:49.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/134288650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[26714]: from='client.56792 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:50.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:14:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:14:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:14:50.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:50 smithi181 ceph-mon[30237]: pgmap v260: 331 pgs: 40 unknown, 64 creating+peering, 227 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:50.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:50 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-21"}]': finished 2024-04-27T01:14:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:50 smithi181 ceph-mon[30237]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:50 smithi181 ceph-mon[30237]: osdmap e202: 8 total, 8 up, 8 in 2024-04-27T01:14:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-21", "mode": "writeback"}]: dispatch 2024-04-27T01:14:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:50 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-21", "mode": "writeback"}]: dispatch 2024-04-27T01:14:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:50 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:50.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[26714]: pgmap v260: 331 pgs: 40 unknown, 64 creating+peering, 227 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:50.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-21"}]': finished 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[26714]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[26714]: osdmap e202: 8 total, 8 up, 8 in 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-21", "mode": "writeback"}]: dispatch 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-21", "mode": "writeback"}]: dispatch 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[37680]: pgmap v260: 331 pgs: 40 unknown, 64 creating+peering, 227 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-21"}]': finished 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[37680]: from='client.56792 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[37680]: osdmap e202: 8 total, 8 up, 8 in 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-21", "mode": "writeback"}]: dispatch 2024-04-27T01:14:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-21", "mode": "writeback"}]: dispatch 2024-04-27T01:14:50.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:14:50.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:14:50.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:51.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:51 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:51.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:51 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-21", "mode": "writeback"}]': finished 2024-04-27T01:14:51.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:51 smithi181 ceph-mon[30237]: osdmap e203: 8 total, 8 up, 8 in 2024-04-27T01:14:51.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/949835838' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:51.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:51 smithi181 ceph-mon[30237]: from='client.56798 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:51.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:51.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:51 smithi181 ceph-mon[30237]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:51.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:51.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:51 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:51.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:51.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:51.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-21", "mode": "writeback"}]': finished 2024-04-27T01:14:51.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[37680]: osdmap e203: 8 total, 8 up, 8 in 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/949835838' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[37680]: from='client.56798 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[37680]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-21", "mode": "writeback"}]': finished 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[26714]: osdmap e203: 8 total, 8 up, 8 in 2024-04-27T01:14:51.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/949835838' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:51.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[26714]: from='client.56798 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:51.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:51.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[26714]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:51.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:51.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:14:51.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:52.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:14:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-mon[30237]: pgmap v263: 355 pgs: 64 unknown, 64 creating+peering, 227 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-27T01:14:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-mon[30237]: from='client.56798 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-mon[30237]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]': finished 2024-04-27T01:14:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-mon[30237]: osdmap e204: 8 total, 8 up, 8 in 2024-04-27T01:14:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-mon[30237]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[37680]: pgmap v263: 355 pgs: 64 unknown, 64 creating+peering, 227 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-27T01:14:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[37680]: from='client.56798 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[37680]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]': finished 2024-04-27T01:14:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[37680]: osdmap e204: 8 total, 8 up, 8 in 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[37680]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[26714]: pgmap v263: 355 pgs: 64 unknown, 64 creating+peering, 227 active+clean; 586 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[26714]: from='client.56798 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi162-74325-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[26714]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi162-74307-32"}]': finished 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[26714]: osdmap e204: 8 total, 8 up, 8 in 2024-04-27T01:14:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1972006942' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:52.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:52.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[26714]: from='client.49039 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]: dispatch 2024-04-27T01:14:52.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21"}]: dispatch 2024-04-27T01:14:52.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]': finished 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21"}]': finished 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: osdmap e205: 8 total, 8 up, 8 in 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-74307-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-74307-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:53.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-04-27T01:14:53.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-04-27T01:14:53.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]': finished 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21"}]': finished 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: osdmap e205: 8 total, 8 up, 8 in 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-74307-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-74307-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-04-27T01:14:53.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='client.49039 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi162-74307-32"}]': finished 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-21"}]': finished 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: osdmap e205: 8 total, 8 up, 8 in 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-74307-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-74307-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:14:53.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-04-27T01:14:53.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-04-27T01:14:53.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:54.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:54 smithi181 ceph-mon[30237]: pgmap v266: 323 pgs: 323 active+clean; 586 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-27T01:14:54.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:54 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:54 smithi181 ceph-mon[30237]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-74307-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:54 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-04-27T01:14:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:54 smithi181 ceph-mon[30237]: osdmap e206: 8 total, 8 up, 8 in 2024-04-27T01:14:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-74307-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:54 smithi181 ceph-mon[30237]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-74307-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[37680]: pgmap v266: 323 pgs: 323 active+clean; 586 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-27T01:14:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[37680]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-74307-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[37680]: osdmap e206: 8 total, 8 up, 8 in 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-74307-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[37680]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-74307-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[26714]: pgmap v266: 323 pgs: 323 active+clean; 586 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[26714]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi162-74307-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi162-74405-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[26714]: osdmap e206: 8 total, 8 up, 8 in 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-74307-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[26714]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-74307-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:54.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4238575682' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-27T01:14:55.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:55.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:55 smithi162 ceph-mon[26714]: osdmap e207: 8 total, 8 up, 8 in 2024-04-27T01:14:55.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:55 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:55.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:55.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:55 smithi162 ceph-mon[37680]: osdmap e207: 8 total, 8 up, 8 in 2024-04-27T01:14:55.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:55 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:55.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:55 smithi181 ceph-mon[30237]: osdmap e207: 8 total, 8 up, 8 in 2024-04-27T01:14:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:55 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:56.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[37680]: pgmap v269: 259 pgs: 32 unknown, 227 active+clean; 586 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[37680]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-74307-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-74307-33"}]': finished 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[37680]: osdmap e208: 8 total, 8 up, 8 in 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3856497236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[37680]: from='client.56810 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-74405-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[26714]: pgmap v269: 259 pgs: 32 unknown, 227 active+clean; 586 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[26714]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-74307-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-74307-33"}]': finished 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:56.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[26714]: osdmap e208: 8 total, 8 up, 8 in 2024-04-27T01:14:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3856497236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[26714]: from='client.56810 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:14:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:14:56.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-74405-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:56.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:56 smithi181 ceph-mon[30237]: pgmap v269: 259 pgs: 32 unknown, 227 active+clean; 586 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:14:56.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:56 smithi181 ceph-mon[30237]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi162-74307-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi162-74307-33"}]': finished 2024-04-27T01:14:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:56 smithi181 ceph-mon[30237]: osdmap e208: 8 total, 8 up, 8 in 2024-04-27T01:14:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3856497236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:56 smithi181 ceph-mon[30237]: from='client.56810 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:14:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:14:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-74405-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:57.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[37680]: from='client.56810 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:57.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-74405-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:57.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[37680]: osdmap e209: 8 total, 8 up, 8 in 2024-04-27T01:14:57.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-74405-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:14:57.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:57.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:57.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:57.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[26714]: from='client.56810 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:57.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-74405-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:57.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[26714]: osdmap e209: 8 total, 8 up, 8 in 2024-04-27T01:14:57.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-74405-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:14:57.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:57.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:57.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:57 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:58.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:57 smithi181 ceph-mon[30237]: from='client.56810 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:58.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi162-74405-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:14:58.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:57 smithi181 ceph-mon[30237]: osdmap e209: 8 total, 8 up, 8 in 2024-04-27T01:14:58.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-74405-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:14:58.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:58.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:57 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:14:58.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:57 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:14:58.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: pgmap v272: 268 pgs: 36 unknown, 232 active+clean; 584 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: osdmap e210: 8 total, 8 up, 8 in 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2137288854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-74405-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-74405-2"}]': finished 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-23"}]': finished 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]': finished 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2137288854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:58.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: osdmap e211: 8 total, 8 up, 8 in 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-23", "mode": "writeback"}]: dispatch 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-23", "mode": "writeback"}]: dispatch 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: pgmap v272: 268 pgs: 36 unknown, 232 active+clean; 584 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: osdmap e210: 8 total, 8 up, 8 in 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:58.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2137288854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:58.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-74405-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-74405-2"}]': finished 2024-04-27T01:14:58.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-23"}]': finished 2024-04-27T01:14:58.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]': finished 2024-04-27T01:14:58.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2137288854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:58.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: osdmap e211: 8 total, 8 up, 8 in 2024-04-27T01:14:58.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:58.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:58.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-23", "mode": "writeback"}]: dispatch 2024-04-27T01:14:58.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-23", "mode": "writeback"}]: dispatch 2024-04-27T01:14:59.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: pgmap v272: 268 pgs: 36 unknown, 232 active+clean; 584 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: osdmap e210: 8 total, 8 up, 8 in 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2137288854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi162-74405-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi162-74405-2"}]': finished 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-23"}]': finished 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi162-74307-33"}]': finished 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2137288854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:14:59.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: osdmap e211: 8 total, 8 up, 8 in 2024-04-27T01:14:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1977727979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.56804 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]: dispatch 2024-04-27T01:14:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-23", "mode": "writeback"}]: dispatch 2024-04-27T01:14:59.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-23", "mode": "writeback"}]: dispatch 2024-04-27T01:14:59.342 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAiod=7 2024-04-27T01:14:59.342 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:566253c9:::13:head 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:566253c9:::13:head expected=13:566253c9:::13:head 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=13 expected=13 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:52ea6a34:::10:head expected=13:52ea6a34:::10:head 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=10 expected=10 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:02547ec2:::1:head expected=13:02547ec2:::1:head 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=1 expected=1 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:b29083e3:::5:head expected=13:b29083e3:::5:head 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-04-27T01:14:59.343 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=5 expected=5 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:89d3ae78:::11:head expected=13:89d3ae78:::11:head 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=11 expected=11 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:863748b0:::15:head 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:863748b0:::15:head expected=13:863748b0:::15:head 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=15 expected=15 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:6cac518f:::0:head expected=13:6cac518f:::0:head 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=0 expected=0 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : cursor()=13:62a1935d:::14:head expected=13:62a1935d:::14:head 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-04-27T01:14:59.344 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: : entry=14 expected=14 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (228 ms) 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (108713 ms) 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: split 0/5 -> MIN 13:33333333::::head 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: split 1/5 -> 13:33333333::::head 13:66666666::::head 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: split 2/5 -> 13:66666666::::head 13:99999999::::head 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: split 3/5 -> 13:99999999::::head 13:cccccccc::::head 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: split 4/5 -> 13:cccccccc::::head MAX 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (23107 ms) 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 7 tests from LibRadosList (132659 ms total) 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-04-27T01:14:59.345 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1035 ms) 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo1,foo2,foo3 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo1 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo2 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo3 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo1,foo4,foo5 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo4 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo5 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo1 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset foo6,foo7 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo7 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: foo6 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo4 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo5 2024-04-27T01:14:59.346 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns2:foo7 2024-04-27T01:14:59.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns2:foo6 2024-04-27T01:14:59.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: ns1:foo1 2024-04-27T01:14:59.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo1 2024-04-27T01:14:59.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo2 2024-04-27T01:14:59.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: :foo3 2024-04-27T01:14:59.347 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (33 ms) 2024-04-27T01:14:59.837 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:14:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:14:59.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[26714]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]': finished 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-23", "mode": "writeback"}]': finished 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[26714]: osdmap e212: 8 total, 8 up, 8 in 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3336229938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[26714]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-74307-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:14:59.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:14:59.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[37680]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]': finished 2024-04-27T01:14:59.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-23", "mode": "writeback"}]': finished 2024-04-27T01:14:59.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[37680]: osdmap e212: 8 total, 8 up, 8 in 2024-04-27T01:14:59.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3336229938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:59.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[37680]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:14:59.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:14:59.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:14:59.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:14:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-74307-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:59 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:59 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:59 smithi181 ceph-mon[30237]: from='client.56804 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi162-74307-33"}]': finished 2024-04-27T01:15:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:59 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-23", "mode": "writeback"}]': finished 2024-04-27T01:15:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:59 smithi181 ceph-mon[30237]: osdmap e212: 8 total, 8 up, 8 in 2024-04-27T01:15:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3336229938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:59 smithi181 ceph-mon[30237]: from='client.56828 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:14:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-74307-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:00.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[26714]: pgmap v275: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:00.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[26714]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-74307-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[26714]: osdmap e213: 8 total, 8 up, 8 in 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-74307-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[37680]: pgmap v275: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[37680]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-74307-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[37680]: osdmap e213: 8 total, 8 up, 8 in 2024-04-27T01:15:00.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-74307-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:00.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:15:00.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:00.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:00 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:01.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:00 smithi181 ceph-mon[30237]: pgmap v275: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:01.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:00 smithi181 ceph-mon[30237]: from='client.56828 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:01.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi162-74307-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:01.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:00 smithi181 ceph-mon[30237]: osdmap e213: 8 total, 8 up, 8 in 2024-04-27T01:15:01.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-74307-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:01.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:15:01.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:01.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:00 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:02.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[37680]: pgmap v278: 324 pgs: 64 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-04-27T01:15:02.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-74405-2"}]': finished 2024-04-27T01:15:02.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:02.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[37680]: osdmap e214: 8 total, 8 up, 8 in 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2644379451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[26714]: pgmap v278: 324 pgs: 64 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-74405-2"}]': finished 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[26714]: osdmap e214: 8 total, 8 up, 8 in 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:15:02.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2644379451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:02.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:15:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:15:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:02 smithi181 ceph-mon[30237]: pgmap v278: 324 pgs: 64 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-04-27T01:15:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi162-74405-2"}]': finished 2024-04-27T01:15:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:15:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:02 smithi181 ceph-mon[30237]: osdmap e214: 8 total, 8 up, 8 in 2024-04-27T01:15:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-74405-2"}]: dispatch 2024-04-27T01:15:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23"}]: dispatch 2024-04-27T01:15:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2644379451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-74307-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-74307-34"}]': finished 2024-04-27T01:15:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-74405-2"}]': finished 2024-04-27T01:15:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23"}]': finished 2024-04-27T01:15:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2644379451' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[26714]: osdmap e215: 8 total, 8 up, 8 in 2024-04-27T01:15:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:03.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:03.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-74307-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-74307-34"}]': finished 2024-04-27T01:15:03.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-74405-2"}]': finished 2024-04-27T01:15:03.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23"}]': finished 2024-04-27T01:15:03.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2644379451' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:03.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[37680]: osdmap e215: 8 total, 8 up, 8 in 2024-04-27T01:15:03.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:04.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:03 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:04.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi162-74307-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi162-74307-34"}]': finished 2024-04-27T01:15:04.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2655068969' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi162-74405-2"}]': finished 2024-04-27T01:15:04.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-23"}]': finished 2024-04-27T01:15:04.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2644379451' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:04.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:03 smithi181 ceph-mon[30237]: osdmap e215: 8 total, 8 up, 8 in 2024-04-27T01:15:04.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: pgmap v281: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 584 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T01:15:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: osdmap e216: 8 total, 8 up, 8 in 2024-04-27T01:15:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2271705961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3089225294' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.49105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.49099 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.49105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.49099 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: osdmap e217: 8 total, 8 up, 8 in 2024-04-27T01:15:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3089225294' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-74405-3","pool2":"test-rados-api-smithi162-74405-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-27T01:15:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[37680]: from='client.49099 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-74405-3","pool2":"test-rados-api-smithi162-74405-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: pgmap v281: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 584 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: osdmap e216: 8 total, 8 up, 8 in 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2271705961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3089225294' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.49105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.49099 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.49105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.49099 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: osdmap e217: 8 total, 8 up, 8 in 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:04.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3089225294' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-74405-3","pool2":"test-rados-api-smithi162-74405-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-27T01:15:04.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:04 smithi162 ceph-mon[26714]: from='client.49099 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-74405-3","pool2":"test-rados-api-smithi162-74405-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-27T01:15:05.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: pgmap v281: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 584 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: osdmap e216: 8 total, 8 up, 8 in 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2271705961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3089225294' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.49105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.49099 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.49105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi162-74325-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.49099 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: osdmap e217: 8 total, 8 up, 8 in 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:05.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3089225294' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-74405-3","pool2":"test-rados-api-smithi162-74405-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-27T01:15:05.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:04 smithi181 ceph-mon[30237]: from='client.49099 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-74405-3","pool2":"test-rados-api-smithi162-74405-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-27T01:15:05.119 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-04-27T01:15:05.119 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 1 0 2024-04-27T01:15:05.119 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 10 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 13 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 7 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 14 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 0 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 15 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 11 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 5 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 8 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 6 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 3 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 4 0 2024-04-27T01:15:05.120 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 12 0 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 9 0 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 2 0 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (50 ms) 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 3 tests from LibRadosListEC (1118 ms total) 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2826 ms) 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] 1 test from LibRadosListNP (2826 ms total) 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: 2024-04-27T01:15:05.121 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [----------] Global test environment tear-down 2024-04-27T01:15:05.122 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [==========] 11 tests from 3 test suites ran. (152396 ms total) 2024-04-27T01:15:05.122 INFO:tasks.workunit.client.0.smithi162.stdout: api_list: [ PASSED ] 11 tests. 2024-04-27T01:15:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-74307-34"}]': finished 2024-04-27T01:15:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[37680]: from='client.49099 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-74405-3","pool2":"test-rados-api-smithi162-74405-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-27T01:15:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[37680]: osdmap e218: 8 total, 8 up, 8 in 2024-04-27T01:15:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-74307-34"}]': finished 2024-04-27T01:15:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[26714]: from='client.49099 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-74405-3","pool2":"test-rados-api-smithi162-74405-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-27T01:15:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[26714]: osdmap e218: 8 total, 8 up, 8 in 2024-04-27T01:15:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:06.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:05 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi162-74307-34"}]': finished 2024-04-27T01:15:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:05 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:05 smithi181 ceph-mon[30237]: from='client.49099 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi162-74405-3","pool2":"test-rados-api-smithi162-74405-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-27T01:15:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:05 smithi181 ceph-mon[30237]: osdmap e218: 8 total, 8 up, 8 in 2024-04-27T01:15:06.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-74307-34"}]: dispatch 2024-04-27T01:15:06.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[37680]: pgmap v285: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 263 B/s wr, 2 op/s 2024-04-27T01:15:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-74307-34"}]': finished 2024-04-27T01:15:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[37680]: osdmap e219: 8 total, 8 up, 8 in 2024-04-27T01:15:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-74307-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[26714]: pgmap v285: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 263 B/s wr, 2 op/s 2024-04-27T01:15:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-74307-34"}]': finished 2024-04-27T01:15:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[26714]: osdmap e219: 8 total, 8 up, 8 in 2024-04-27T01:15:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:06.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-74307-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:07.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:06 smithi181 ceph-mon[30237]: pgmap v285: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 263 B/s wr, 2 op/s 2024-04-27T01:15:07.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:06 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:07.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4016968444' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi162-74307-34"}]': finished 2024-04-27T01:15:07.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:06 smithi181 ceph-mon[30237]: osdmap e219: 8 total, 8 up, 8 in 2024-04-27T01:15:07.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:07.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:06 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:07.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:07.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:07.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-74307-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-74307-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:08 smithi181 ceph-mon[30237]: osdmap e220: 8 total, 8 up, 8 in 2024-04-27T01:15:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi162-74307-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:08 smithi181 ceph-mon[30237]: pgmap v288: 292 pgs: 13 unknown, 279 active+clean; 584 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:15:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-74307-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[37680]: osdmap e220: 8 total, 8 up, 8 in 2024-04-27T01:15:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:08.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi162-74307-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:08.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:08.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[37680]: pgmap v288: 292 pgs: 13 unknown, 279 active+clean; 584 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:15:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi162-74307-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[26714]: osdmap e220: 8 total, 8 up, 8 in 2024-04-27T01:15:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi162-74307-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:08 smithi162 ceph-mon[26714]: pgmap v288: 292 pgs: 13 unknown, 279 active+clean; 584 KiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:15:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-25"}]': finished 2024-04-27T01:15:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-25", "mode": "writeback"}]: dispatch 2024-04-27T01:15:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:09 smithi181 ceph-mon[30237]: osdmap e221: 8 total, 8 up, 8 in 2024-04-27T01:15:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-25", "mode": "writeback"}]: dispatch 2024-04-27T01:15:09.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-25"}]': finished 2024-04-27T01:15:09.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-25", "mode": "writeback"}]: dispatch 2024-04-27T01:15:09.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:09 smithi162 ceph-mon[37680]: osdmap e221: 8 total, 8 up, 8 in 2024-04-27T01:15:09.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-25", "mode": "writeback"}]: dispatch 2024-04-27T01:15:09.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-25"}]': finished 2024-04-27T01:15:09.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-25", "mode": "writeback"}]: dispatch 2024-04-27T01:15:09.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:09 smithi162 ceph-mon[26714]: osdmap e221: 8 total, 8 up, 8 in 2024-04-27T01:15:09.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-25", "mode": "writeback"}]: dispatch 2024-04-27T01:15:10.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:15:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:15:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:15:10.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:10 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:10.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi162-74307-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-74307-35"}]': finished 2024-04-27T01:15:10.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-25", "mode": "writeback"}]': finished 2024-04-27T01:15:10.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:10 smithi181 ceph-mon[30237]: osdmap e222: 8 total, 8 up, 8 in 2024-04-27T01:15:10.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:10 smithi181 ceph-mon[30237]: pgmap v291: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 536 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:10.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:10.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:10.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:10.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi162-74307-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-74307-35"}]': finished 2024-04-27T01:15:10.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-25", "mode": "writeback"}]': finished 2024-04-27T01:15:10.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[37680]: osdmap e222: 8 total, 8 up, 8 in 2024-04-27T01:15:10.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[37680]: pgmap v291: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 536 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:10.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:10.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:10.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:10.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi162-74307-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi162-74307-35"}]': finished 2024-04-27T01:15:10.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-25", "mode": "writeback"}]': finished 2024-04-27T01:15:10.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[26714]: osdmap e222: 8 total, 8 up, 8 in 2024-04-27T01:15:10.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[26714]: pgmap v291: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 536 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:10.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:10.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:11.491 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:11.491 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[37680]: osdmap e223: 8 total, 8 up, 8 in 2024-04-27T01:15:11.491 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/703151159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-74325-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[37680]: from='client.56864 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-74325-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[26714]: osdmap e223: 8 total, 8 up, 8 in 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/703151159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-74325-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:11.492 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[26714]: from='client.56864 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-74325-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:11.493 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:15:11.493 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:11.493 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:11 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:11.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:11 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:11.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:11 smithi181 ceph-mon[30237]: osdmap e223: 8 total, 8 up, 8 in 2024-04-27T01:15:11.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:11.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/703151159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-74325-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:11.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:11 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25"}]: dispatch 2024-04-27T01:15:11.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:11 smithi181 ceph-mon[30237]: from='client.56864 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-74325-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:11.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:11 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:15:11.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:11 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:11.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:11 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:12.625 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:15:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25"}]': finished 2024-04-27T01:15:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: from='client.56864 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-74325-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: osdmap e224: 8 total, 8 up, 8 in 2024-04-27T01:15:13.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:13.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: pgmap v294: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 536 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:13.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:15:13.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:15:13.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:13.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-74307-35"}]': finished 2024-04-27T01:15:13.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: osdmap e225: 8 total, 8 up, 8 in 2024-04-27T01:15:13.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25"}]': finished 2024-04-27T01:15:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: from='client.56864 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-74325-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: osdmap e224: 8 total, 8 up, 8 in 2024-04-27T01:15:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:13.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: pgmap v294: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 536 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:13.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:15:13.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:15:13.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:13.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-74307-35"}]': finished 2024-04-27T01:15:13.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: osdmap e225: 8 total, 8 up, 8 in 2024-04-27T01:15:13.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:13.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:13.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-25"}]': finished 2024-04-27T01:15:13.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: from='client.56864 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi162-74325-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:13.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: osdmap e224: 8 total, 8 up, 8 in 2024-04-27T01:15:13.090 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:13.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: pgmap v294: 260 pgs: 32 unknown, 228 active+clean; 584 KiB data, 536 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:13.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:15:13.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:15:13.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:13.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi162-74307-35"}]': finished 2024-04-27T01:15:13.091 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: osdmap e225: 8 total, 8 up, 8 in 2024-04-27T01:15:13.092 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-74307-35"}]: dispatch 2024-04-27T01:15:14.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:14.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-74307-35"}]': finished 2024-04-27T01:15:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: osdmap e226: 8 total, 8 up, 8 in 2024-04-27T01:15:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/852501667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-74325-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: from='client.56870 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-74325-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:14.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-74307-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:14.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:14.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-74307-35"}]': finished 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: osdmap e226: 8 total, 8 up, 8 in 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/852501667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-74325-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: from='client.56870 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-74325-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-74307-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:14.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1278405854' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi162-74307-35"}]': finished 2024-04-27T01:15:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: osdmap e226: 8 total, 8 up, 8 in 2024-04-27T01:15:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/852501667' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-74325-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: from='client.56870 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-74325-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:14.089 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-74307-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:15.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:14 smithi181 ceph-mon[30237]: pgmap v297: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:15.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:14 smithi181 ceph-mon[30237]: from='client.56870 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-74325-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:15.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:14 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:15.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-74307-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:15.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:14 smithi181 ceph-mon[30237]: osdmap e227: 8 total, 8 up, 8 in 2024-04-27T01:15:15.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-74307-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:15.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[37680]: pgmap v297: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:15.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[37680]: from='client.56870 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-74325-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:15.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:15.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-74307-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:15.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[37680]: osdmap e227: 8 total, 8 up, 8 in 2024-04-27T01:15:15.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-74307-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:15.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[26714]: pgmap v297: 260 pgs: 64 unknown, 196 active+clean; 584 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:15.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[26714]: from='client.56870 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi162-74325-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:15.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:15.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi162-74307-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:15.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[26714]: osdmap e227: 8 total, 8 up, 8 in 2024-04-27T01:15:15.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-74307-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:16.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:16 smithi181 ceph-mon[30237]: pgmap v300: 228 pgs: 32 unknown, 196 active+clean; 584 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:16.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:16 smithi181 ceph-mon[30237]: osdmap e228: 8 total, 8 up, 8 in 2024-04-27T01:15:16.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:16 smithi181 ceph-mon[30237]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:16.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:16 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:16.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:16 smithi181 ceph-mon[30237]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:16.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:16 smithi181 ceph-mon[30237]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:16.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[37680]: pgmap v300: 228 pgs: 32 unknown, 196 active+clean; 584 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:16.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[37680]: osdmap e228: 8 total, 8 up, 8 in 2024-04-27T01:15:16.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[37680]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:16.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:16.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[37680]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:16.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[37680]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:16.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[26714]: pgmap v300: 228 pgs: 32 unknown, 196 active+clean; 584 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:16.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[26714]: osdmap e228: 8 total, 8 up, 8 in 2024-04-27T01:15:16.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[26714]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:16.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:16.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[26714]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:16.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:16.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:16 smithi162 ceph-mon[26714]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:17.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-74307-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-74307-36"}]': finished 2024-04-27T01:15:17.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:17 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:17 smithi181 ceph-mon[30237]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:17 smithi181 ceph-mon[30237]: osdmap e229: 8 total, 8 up, 8 in 2024-04-27T01:15:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:17 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:17.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:17 smithi181 ceph-mon[30237]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:17.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-74307-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-74307-36"}]': finished 2024-04-27T01:15:17.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:17.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[37680]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:17.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[37680]: osdmap e229: 8 total, 8 up, 8 in 2024-04-27T01:15:17.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:17.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:17.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:17.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[37680]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:17.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi162-74307-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi162-74307-36"}]': finished 2024-04-27T01:15:17.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:17.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[26714]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:17.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[26714]: osdmap e229: 8 total, 8 up, 8 in 2024-04-27T01:15:17.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:17.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:17.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:17.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:17 smithi162 ceph-mon[26714]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:18.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-27"}]': finished 2024-04-27T01:15:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-27", "mode": "writeback"}]: dispatch 2024-04-27T01:15:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: pgmap v303: 236 pgs: 6 creating+peering, 15 unknown, 215 active+clean; 584 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:15:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: osdmap e230: 8 total, 8 up, 8 in 2024-04-27T01:15:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-27", "mode": "writeback"}]: dispatch 2024-04-27T01:15:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-46"}]': finished 2024-04-27T01:15:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-27", "mode": "writeback"}]': finished 2024-04-27T01:15:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: osdmap e231: 8 total, 8 up, 8 in 2024-04-27T01:15:18.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:18.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-27"}]': finished 2024-04-27T01:15:18.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-27", "mode": "writeback"}]: dispatch 2024-04-27T01:15:18.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: pgmap v303: 236 pgs: 6 creating+peering, 15 unknown, 215 active+clean; 584 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:15:18.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: osdmap e230: 8 total, 8 up, 8 in 2024-04-27T01:15:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-27", "mode": "writeback"}]: dispatch 2024-04-27T01:15:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-46"}]': finished 2024-04-27T01:15:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-27", "mode": "writeback"}]': finished 2024-04-27T01:15:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: osdmap e231: 8 total, 8 up, 8 in 2024-04-27T01:15:18.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:18.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-27"}]': finished 2024-04-27T01:15:18.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-27", "mode": "writeback"}]: dispatch 2024-04-27T01:15:18.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: pgmap v303: 236 pgs: 6 creating+peering, 15 unknown, 215 active+clean; 584 KiB data, 537 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:15:18.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: osdmap e230: 8 total, 8 up, 8 in 2024-04-27T01:15:18.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-27", "mode": "writeback"}]: dispatch 2024-04-27T01:15:18.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:18.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:18.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-46"}]': finished 2024-04-27T01:15:18.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-27", "mode": "writeback"}]': finished 2024-04-27T01:15:18.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: osdmap e231: 8 total, 8 up, 8 in 2024-04-27T01:15:18.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:20.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:15:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:15:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:15:20.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-74307-36"}]': finished 2024-04-27T01:15:20.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:20 smithi181 ceph-mon[30237]: pgmap v305: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:15:20.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:20 smithi181 ceph-mon[30237]: osdmap e232: 8 total, 8 up, 8 in 2024-04-27T01:15:20.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:20.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:20.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:20 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:20.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:20 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:20.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-74307-36"}]': finished 2024-04-27T01:15:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[37680]: pgmap v305: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:15:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[37680]: osdmap e232: 8 total, 8 up, 8 in 2024-04-27T01:15:20.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:20.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:20.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:20.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:20.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi162-74307-36"}]': finished 2024-04-27T01:15:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[26714]: pgmap v305: 236 pgs: 8 unknown, 228 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:15:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[26714]: osdmap e232: 8 total, 8 up, 8 in 2024-04-27T01:15:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-74307-36"}]: dispatch 2024-04-27T01:15:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:20.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:20.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:20.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:21.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-74307-36"}]': finished 2024-04-27T01:15:21.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:21.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: osdmap e233: 8 total, 8 up, 8 in 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27"}]': finished 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]': finished 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:21.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: osdmap e234: 8 total, 8 up, 8 in 2024-04-27T01:15:21.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-74307-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:21.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:21 smithi181 ceph-mon[30237]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-74307-36"}]': finished 2024-04-27T01:15:21.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: osdmap e233: 8 total, 8 up, 8 in 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27"}]': finished 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]': finished 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: osdmap e234: 8 total, 8 up, 8 in 2024-04-27T01:15:21.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-74307-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[37680]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2534796105' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi162-74307-36"}]': finished 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: osdmap e233: 8 total, 8 up, 8 in 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27"}]: dispatch 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:21.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:21.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:21.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:21.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-27"}]': finished 2024-04-27T01:15:21.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-46"}]': finished 2024-04-27T01:15:21.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:21.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: osdmap e234: 8 total, 8 up, 8 in 2024-04-27T01:15:21.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-74307-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:21.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2961439983' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:21.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:21 smithi162 ceph-mon[26714]: from='client.56882 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]: dispatch 2024-04-27T01:15:22.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:15:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:15:22.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:22 smithi181 ceph-mon[30237]: pgmap v308: 228 pgs: 228 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-27T01:15:22.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:22 smithi181 ceph-mon[30237]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]': finished 2024-04-27T01:15:22.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:22 smithi162 ceph-mon[37680]: pgmap v308: 228 pgs: 228 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-27T01:15:22.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:22 smithi162 ceph-mon[37680]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]': finished 2024-04-27T01:15:22.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:22 smithi162 ceph-mon[26714]: pgmap v308: 228 pgs: 228 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-27T01:15:22.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:22 smithi162 ceph-mon[26714]: from='client.56882 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-46"}]': finished 2024-04-27T01:15:23.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:23.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: osdmap e235: 8 total, 8 up, 8 in 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-74307-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-74307-37"}]': finished 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: osdmap e236: 8 total, 8 up, 8 in 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:23.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:23 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:23.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:23.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: osdmap e235: 8 total, 8 up, 8 in 2024-04-27T01:15:23.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-74307-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-74307-37"}]': finished 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: osdmap e236: 8 total, 8 up, 8 in 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:23.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: osdmap e235: 8 total, 8 up, 8 in 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi162-74307-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi162-74307-37"}]': finished 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi162-74325-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: osdmap e236: 8 total, 8 up, 8 in 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:23.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:23 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:24.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:24 smithi181 ceph-mon[30237]: pgmap v311: 196 pgs: 196 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:24.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:24 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:24.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:24 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:24.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:24 smithi181 ceph-mon[30237]: osdmap e237: 8 total, 8 up, 8 in 2024-04-27T01:15:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:24 smithi162 ceph-mon[37680]: pgmap v311: 196 pgs: 196 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:24 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:24 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:24 smithi162 ceph-mon[37680]: osdmap e237: 8 total, 8 up, 8 in 2024-04-27T01:15:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:24 smithi162 ceph-mon[26714]: pgmap v311: 196 pgs: 196 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:24 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:24 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:24 smithi162 ceph-mon[26714]: osdmap e237: 8 total, 8 up, 8 in 2024-04-27T01:15:26.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:26 smithi181 ceph-mon[30237]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-47"}]': finished 2024-04-27T01:15:26.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:26 smithi181 ceph-mon[30237]: osdmap e238: 8 total, 8 up, 8 in 2024-04-27T01:15:26.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:26.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:26 smithi181 ceph-mon[30237]: pgmap v315: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-27T01:15:26.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:26 smithi162 ceph-mon[37680]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-47"}]': finished 2024-04-27T01:15:26.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:26 smithi162 ceph-mon[37680]: osdmap e238: 8 total, 8 up, 8 in 2024-04-27T01:15:26.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:26.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:26 smithi162 ceph-mon[37680]: pgmap v315: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-27T01:15:26.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:26 smithi162 ceph-mon[26714]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi162-74325-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi162-74325-47"}]': finished 2024-04-27T01:15:26.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:26 smithi162 ceph-mon[26714]: osdmap e238: 8 total, 8 up, 8 in 2024-04-27T01:15:26.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:26.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:26 smithi162 ceph-mon[26714]: pgmap v315: 236 pgs: 40 unknown, 196 active+clean; 584 KiB data, 538 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-27T01:15:27.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37"}]': finished 2024-04-27T01:15:27.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:27 smithi181 ceph-mon[30237]: osdmap e239: 8 total, 8 up, 8 in 2024-04-27T01:15:27.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:27.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:27.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:27 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:27.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37"}]': finished 2024-04-27T01:15:27.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:27 smithi162 ceph-mon[37680]: osdmap e239: 8 total, 8 up, 8 in 2024-04-27T01:15:27.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:27.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:27.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:27 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:27.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi162-74307-37"}]': finished 2024-04-27T01:15:27.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:27 smithi162 ceph-mon[26714]: osdmap e239: 8 total, 8 up, 8 in 2024-04-27T01:15:27.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-74307-37"}]: dispatch 2024-04-27T01:15:27.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:27.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:27 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:28.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-74307-37"}]': finished 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: osdmap e240: 8 total, 8 up, 8 in 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-74307-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-74307-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: pgmap v318: 228 pgs: 11 unknown, 217 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-29"}]': finished 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]': finished 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-74307-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: osdmap e241: 8 total, 8 up, 8 in 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-29", "mode": "writeback"}]: dispatch 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-29", "mode": "writeback"}]: dispatch 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-74307-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-74307-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:28 smithi181 ceph-mon[30237]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-74307-37"}]': finished 2024-04-27T01:15:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: osdmap e240: 8 total, 8 up, 8 in 2024-04-27T01:15:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-74307-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-74307-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: pgmap v318: 228 pgs: 11 unknown, 217 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-29"}]': finished 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]': finished 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-74307-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: osdmap e241: 8 total, 8 up, 8 in 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-29", "mode": "writeback"}]: dispatch 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3296625' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi162-74307-37"}]': finished 2024-04-27T01:15:28.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: osdmap e240: 8 total, 8 up, 8 in 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-74307-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-74307-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:28.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: pgmap v318: 228 pgs: 11 unknown, 217 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-04-27T01:15:28.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-29"}]': finished 2024-04-27T01:15:28.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi162-74325-47"}]': finished 2024-04-27T01:15:28.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi162-74307-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:28.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: osdmap e241: 8 total, 8 up, 8 in 2024-04-27T01:15:28.591 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-29", "mode": "writeback"}]: dispatch 2024-04-27T01:15:28.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-29", "mode": "writeback"}]: dispatch 2024-04-27T01:15:28.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-74307-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-74307-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.591 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[37680]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-29", "mode": "writeback"}]: dispatch 2024-04-27T01:15:28.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-74307-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-74307-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:28.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/816636838' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:28.592 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:28 smithi162 ceph-mon[26714]: from='client.56894 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]: dispatch 2024-04-27T01:15:29.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:29 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:29.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:29 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-29", "mode": "writeback"}]': finished 2024-04-27T01:15:29.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:29 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:29.729 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:29 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-29", "mode": "writeback"}]': finished 2024-04-27T01:15:29.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:29 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:29.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:29 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-29", "mode": "writeback"}]': finished 2024-04-27T01:15:30.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:15:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:15:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:15:30.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]': finished 2024-04-27T01:15:30.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: pgmap v321: 228 pgs: 228 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: osdmap e242: 8 total, 8 up, 8 in 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-74325-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-74325-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-74307-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-74307-38"}]': finished 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-74325-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-74325-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: osdmap e243: 8 total, 8 up, 8 in 2024-04-27T01:15:30.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:30 smithi181 ceph-mon[30237]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-74325-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]': finished 2024-04-27T01:15:30.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: pgmap v321: 228 pgs: 228 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: osdmap e242: 8 total, 8 up, 8 in 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-74325-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-74325-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-74307-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-74307-38"}]': finished 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-74325-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-74325-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: osdmap e243: 8 total, 8 up, 8 in 2024-04-27T01:15:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[26714]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-74325-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.56894 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi162-74325-47"}]': finished 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: pgmap v321: 228 pgs: 228 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: osdmap e242: 8 total, 8 up, 8 in 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-74325-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-74325-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi162-74307-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi162-74307-38"}]': finished 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi162-74325-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:30.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-74325-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:30.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: osdmap e243: 8 total, 8 up, 8 in 2024-04-27T01:15:30.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:30 smithi162 ceph-mon[37680]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-74325-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:32.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:32 smithi181 ceph-mon[30237]: pgmap v324: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:32.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:32 smithi181 ceph-mon[30237]: osdmap e244: 8 total, 8 up, 8 in 2024-04-27T01:15:32.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:15:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:15:32.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:32 smithi162 ceph-mon[37680]: pgmap v324: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:32.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:32 smithi162 ceph-mon[37680]: osdmap e244: 8 total, 8 up, 8 in 2024-04-27T01:15:32.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:32 smithi162 ceph-mon[26714]: pgmap v324: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 559 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:32.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:32 smithi162 ceph-mon[26714]: osdmap e244: 8 total, 8 up, 8 in 2024-04-27T01:15:33.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:33 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:33.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:33 smithi181 ceph-mon[30237]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-74325-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-74325-48"}]': finished 2024-04-27T01:15:33.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:33 smithi181 ceph-mon[30237]: osdmap e245: 8 total, 8 up, 8 in 2024-04-27T01:15:33.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:33.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:33 smithi181 ceph-mon[30237]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:33.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:33.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:33 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:33.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:33.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:33.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[37680]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-74325-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-74325-48"}]': finished 2024-04-27T01:15:33.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[37680]: osdmap e245: 8 total, 8 up, 8 in 2024-04-27T01:15:33.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[37680]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[26714]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi162-74325-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi162-74325-48"}]': finished 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[26714]: osdmap e245: 8 total, 8 up, 8 in 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[26714]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:33.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:33.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:34.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]': finished 2024-04-27T01:15:34.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:34.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: osdmap e246: 8 total, 8 up, 8 in 2024-04-27T01:15:34.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:34.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:34.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:34.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:34.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: pgmap v327: 236 pgs: 4 creating+peering, 1 active+clean+snaptrim, 4 unknown, 227 active+clean; 8.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-27T01:15:34.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:34.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29"}]': finished 2024-04-27T01:15:34.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]': finished 2024-04-27T01:15:34.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: osdmap e247: 8 total, 8 up, 8 in 2024-04-27T01:15:34.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:34.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]': finished 2024-04-27T01:15:34.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:34.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: osdmap e246: 8 total, 8 up, 8 in 2024-04-27T01:15:34.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:34.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:34.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: pgmap v327: 236 pgs: 4 creating+peering, 1 active+clean+snaptrim, 4 unknown, 227 active+clean; 8.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29"}]': finished 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]': finished 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: osdmap e247: 8 total, 8 up, 8 in 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi162-74307-38"}]': finished 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: osdmap e246: 8 total, 8 up, 8 in 2024-04-27T01:15:34.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:34.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/500004879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:34.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29"}]: dispatch 2024-04-27T01:15:34.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: from='client.56900 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]: dispatch 2024-04-27T01:15:34.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: pgmap v327: 236 pgs: 4 creating+peering, 1 active+clean+snaptrim, 4 unknown, 227 active+clean; 8.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-27T01:15:34.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:34.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-29"}]': finished 2024-04-27T01:15:34.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: from='client.56900 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi162-74307-38"}]': finished 2024-04-27T01:15:34.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: osdmap e247: 8 total, 8 up, 8 in 2024-04-27T01:15:34.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:35 smithi181 ceph-mon[30237]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:35.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:35.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-74307-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:35.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:35 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:35.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:35 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:35.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:35 smithi181 ceph-mon[30237]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]': finished 2024-04-27T01:15:35.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-74307-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:35.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[37680]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-74307-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[37680]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]': finished 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-74307-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[26714]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-74307-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[26714]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi162-74325-48"}]': finished 2024-04-27T01:15:35.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi162-74307-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:35.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3239685088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:36 smithi181 ceph-mon[30237]: osdmap e248: 8 total, 8 up, 8 in 2024-04-27T01:15:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-74307-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:36 smithi181 ceph-mon[30237]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:36 smithi181 ceph-mon[30237]: pgmap v330: 196 pgs: 1 active+clean+snaptrim, 195 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:36 smithi181 ceph-mon[30237]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]': finished 2024-04-27T01:15:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:36 smithi162 ceph-mon[37680]: osdmap e248: 8 total, 8 up, 8 in 2024-04-27T01:15:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-74307-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:36 smithi162 ceph-mon[37680]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:36.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:36 smithi162 ceph-mon[37680]: pgmap v330: 196 pgs: 1 active+clean+snaptrim, 195 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:36.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:36 smithi162 ceph-mon[37680]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]': finished 2024-04-27T01:15:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:36 smithi162 ceph-mon[26714]: osdmap e248: 8 total, 8 up, 8 in 2024-04-27T01:15:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-74307-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:36 smithi162 ceph-mon[26714]: from='client.49156 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]: dispatch 2024-04-27T01:15:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:36 smithi162 ceph-mon[26714]: pgmap v330: 196 pgs: 1 active+clean+snaptrim, 195 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:36 smithi162 ceph-mon[26714]: from='client.49156 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi162-74325-48"}]': finished 2024-04-27T01:15:37.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: osdmap e249: 8 total, 8 up, 8 in 2024-04-27T01:15:37.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-74325-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-74325-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-74307-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-74307-39"}]': finished 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-74325-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: osdmap e250: 8 total, 8 up, 8 in 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-74325-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-74325-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:37.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:37 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:37.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: osdmap e249: 8 total, 8 up, 8 in 2024-04-27T01:15:37.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-74325-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-74325-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-74307-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-74307-39"}]': finished 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-74325-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: osdmap e250: 8 total, 8 up, 8 in 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-74325-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-74325-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: osdmap e249: 8 total, 8 up, 8 in 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-74325-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-74325-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:37.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi162-74307-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi162-74307-39"}]': finished 2024-04-27T01:15:37.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:37.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi162-74325-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:37.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: osdmap e250: 8 total, 8 up, 8 in 2024-04-27T01:15:37.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-74325-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-74325-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:37.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:37.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:37 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:38.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:38 smithi181 ceph-mon[30237]: pgmap v333: 236 pgs: 4 creating+activating, 17 creating+peering, 19 unknown, 196 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:38.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:38 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:38.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:38 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:38.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:38 smithi162 ceph-mon[37680]: pgmap v333: 236 pgs: 4 creating+activating, 17 creating+peering, 19 unknown, 196 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:38.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:38 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:38.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:38 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:38 smithi162 ceph-mon[26714]: pgmap v333: 236 pgs: 4 creating+activating, 17 creating+peering, 19 unknown, 196 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:38 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:38.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:38 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:39.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:39.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[37680]: osdmap e251: 8 total, 8 up, 8 in 2024-04-27T01:15:39.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:39.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[37680]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-74325-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-74325-49"}]': finished 2024-04-27T01:15:39.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-31"}]': finished 2024-04-27T01:15:39.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[37680]: osdmap e252: 8 total, 8 up, 8 in 2024-04-27T01:15:39.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-31", "mode": "writeback"}]: dispatch 2024-04-27T01:15:39.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-31", "mode": "writeback"}]: dispatch 2024-04-27T01:15:39.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:39.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:39.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[26714]: osdmap e251: 8 total, 8 up, 8 in 2024-04-27T01:15:39.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:39.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[26714]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-74325-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-74325-49"}]': finished 2024-04-27T01:15:39.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-31"}]': finished 2024-04-27T01:15:39.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[26714]: osdmap e252: 8 total, 8 up, 8 in 2024-04-27T01:15:39.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-31", "mode": "writeback"}]: dispatch 2024-04-27T01:15:39.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-31", "mode": "writeback"}]: dispatch 2024-04-27T01:15:39.730 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:39.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:39.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:39 smithi181 ceph-mon[30237]: osdmap e251: 8 total, 8 up, 8 in 2024-04-27T01:15:39.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:39.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:39 smithi181 ceph-mon[30237]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi162-74325-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi162-74325-49"}]': finished 2024-04-27T01:15:39.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-31"}]': finished 2024-04-27T01:15:39.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:39 smithi181 ceph-mon[30237]: osdmap e252: 8 total, 8 up, 8 in 2024-04-27T01:15:39.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-31", "mode": "writeback"}]: dispatch 2024-04-27T01:15:39.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-31", "mode": "writeback"}]: dispatch 2024-04-27T01:15:39.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:40.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:15:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:15:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:15:40.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:40 smithi181 ceph-mon[30237]: pgmap v336: 236 pgs: 4 creating+activating, 17 creating+peering, 8 unknown, 207 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-04-27T01:15:40.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:40 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:40.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-31", "mode": "writeback"}]': finished 2024-04-27T01:15:40.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-74307-39"}]': finished 2024-04-27T01:15:40.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:40 smithi181 ceph-mon[30237]: osdmap e253: 8 total, 8 up, 8 in 2024-04-27T01:15:40.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:40.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:40.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:40.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[26714]: pgmap v336: 236 pgs: 4 creating+activating, 17 creating+peering, 8 unknown, 207 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-31", "mode": "writeback"}]': finished 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-74307-39"}]': finished 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[26714]: osdmap e253: 8 total, 8 up, 8 in 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[37680]: pgmap v336: 236 pgs: 4 creating+activating, 17 creating+peering, 8 unknown, 207 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-31", "mode": "writeback"}]': finished 2024-04-27T01:15:40.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi162-74307-39"}]': finished 2024-04-27T01:15:40.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[37680]: osdmap e253: 8 total, 8 up, 8 in 2024-04-27T01:15:40.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-74307-39"}]: dispatch 2024-04-27T01:15:40.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:40.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:42.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:15:42.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-74307-39"}]': finished 2024-04-27T01:15:42.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:42.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: osdmap e254: 8 total, 8 up, 8 in 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-74307-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-74307-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:42.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:42 smithi181 ceph-mon[30237]: pgmap v339: 228 pgs: 4 creating+activating, 17 creating+peering, 207 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:42.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-74307-39"}]': finished 2024-04-27T01:15:42.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: osdmap e254: 8 total, 8 up, 8 in 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-74307-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-74307-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:42.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[26714]: pgmap v339: 228 pgs: 4 creating+activating, 17 creating+peering, 207 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/238385804' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi162-74307-39"}]': finished 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: osdmap e254: 8 total, 8 up, 8 in 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31"}]: dispatch 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:42.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-74307-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:42.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-74307-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:42.590 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:42 smithi162 ceph-mon[37680]: pgmap v339: 228 pgs: 4 creating+activating, 17 creating+peering, 207 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:43 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:43 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31"}]': finished 2024-04-27T01:15:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:43 smithi181 ceph-mon[30237]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]': finished 2024-04-27T01:15:43.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:43 smithi181 ceph-mon[30237]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-74307-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:43.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:43 smithi181 ceph-mon[30237]: osdmap e255: 8 total, 8 up, 8 in 2024-04-27T01:15:43.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:43.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:43 smithi181 ceph-mon[30237]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:43.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-74307-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:43.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:43 smithi181 ceph-mon[30237]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-74307-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:43.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:43.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:43.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31"}]': finished 2024-04-27T01:15:43.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[26714]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]': finished 2024-04-27T01:15:43.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[26714]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-74307-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:43.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[26714]: osdmap e255: 8 total, 8 up, 8 in 2024-04-27T01:15:43.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:43.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[26714]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:43.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-74307-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:43.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[26714]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-74307-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:43.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:43.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:43.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-31"}]': finished 2024-04-27T01:15:43.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[37680]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi162-74325-49"}]': finished 2024-04-27T01:15:43.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[37680]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi162-74307-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:43.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[37680]: osdmap e255: 8 total, 8 up, 8 in 2024-04-27T01:15:43.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/900175183' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:43.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[37680]: from='client.56915 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]: dispatch 2024-04-27T01:15:43.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-74307-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:43.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[37680]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-74307-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:43.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:44.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]': finished 2024-04-27T01:15:44.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: pgmap v342: 196 pgs: 196 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: osdmap e256: 8 total, 8 up, 8 in 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-74325-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-74325-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-74307-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-74307-40"}]': finished 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-74325-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: osdmap e257: 8 total, 8 up, 8 in 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-74325-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-74325-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:44.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:44 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]': finished 2024-04-27T01:15:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: pgmap v342: 196 pgs: 196 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: osdmap e256: 8 total, 8 up, 8 in 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-74325-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-74325-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-74307-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-74307-40"}]': finished 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-74325-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: osdmap e257: 8 total, 8 up, 8 in 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-74325-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-74325-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.56915 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi162-74325-49"}]': finished 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: pgmap v342: 196 pgs: 196 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: osdmap e256: 8 total, 8 up, 8 in 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-74325-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-74325-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi162-74307-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi162-74307-40"}]': finished 2024-04-27T01:15:44.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi162-74325-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:44.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: osdmap e257: 8 total, 8 up, 8 in 2024-04-27T01:15:44.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-74325-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-74325-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:44.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:44.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:44 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:46.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:46 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:46.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:46.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:46 smithi181 ceph-mon[30237]: pgmap v345: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:46 smithi181 ceph-mon[30237]: osdmap e258: 8 total, 8 up, 8 in 2024-04-27T01:15:46.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:46 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:46.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:46 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:46.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:46.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:46 smithi162 ceph-mon[37680]: pgmap v345: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:46 smithi162 ceph-mon[37680]: osdmap e258: 8 total, 8 up, 8 in 2024-04-27T01:15:46.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:46 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:46.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:46 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:46.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:46.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:46 smithi162 ceph-mon[26714]: pgmap v345: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:46.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:46 smithi162 ceph-mon[26714]: osdmap e258: 8 total, 8 up, 8 in 2024-04-27T01:15:46.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:46 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:47 smithi181 ceph-mon[30237]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-74325-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-74325-50"}]': finished 2024-04-27T01:15:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:47 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:47 smithi181 ceph-mon[30237]: osdmap e259: 8 total, 8 up, 8 in 2024-04-27T01:15:47.511 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:47 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:47.512 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:47 smithi181 ceph-mon[30237]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:47.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[37680]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-74325-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-74325-50"}]': finished 2024-04-27T01:15:47.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:47.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[37680]: osdmap e259: 8 total, 8 up, 8 in 2024-04-27T01:15:47.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:47.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:47.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[37680]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[26714]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi162-74325-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi162-74325-50"}]': finished 2024-04-27T01:15:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[26714]: osdmap e259: 8 total, 8 up, 8 in 2024-04-27T01:15:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:47.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:47 smithi162 ceph-mon[26714]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:48.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:48 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-33"}]': finished 2024-04-27T01:15:48.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:48 smithi181 ceph-mon[30237]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]': finished 2024-04-27T01:15:48.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:48 smithi181 ceph-mon[30237]: osdmap e260: 8 total, 8 up, 8 in 2024-04-27T01:15:48.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:48.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-33", "mode": "writeback"}]: dispatch 2024-04-27T01:15:48.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:48 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-33", "mode": "writeback"}]: dispatch 2024-04-27T01:15:48.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:48 smithi181 ceph-mon[30237]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:48.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:48 smithi181 ceph-mon[30237]: pgmap v348: 236 pgs: 5 creating+peering, 13 unknown, 218 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-33"}]': finished 2024-04-27T01:15:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[37680]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]': finished 2024-04-27T01:15:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[37680]: osdmap e260: 8 total, 8 up, 8 in 2024-04-27T01:15:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-33", "mode": "writeback"}]: dispatch 2024-04-27T01:15:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-33", "mode": "writeback"}]: dispatch 2024-04-27T01:15:48.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[37680]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:48.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[37680]: pgmap v348: 236 pgs: 5 creating+peering, 13 unknown, 218 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-33"}]': finished 2024-04-27T01:15:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[26714]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi162-74307-40"}]': finished 2024-04-27T01:15:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[26714]: osdmap e260: 8 total, 8 up, 8 in 2024-04-27T01:15:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3128987702' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-33", "mode": "writeback"}]: dispatch 2024-04-27T01:15:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-33", "mode": "writeback"}]: dispatch 2024-04-27T01:15:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[26714]: from='client.56921 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]: dispatch 2024-04-27T01:15:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:48 smithi162 ceph-mon[26714]: pgmap v348: 236 pgs: 5 creating+peering, 13 unknown, 218 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:49.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:49.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-33", "mode": "writeback"}]': finished 2024-04-27T01:15:49.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]': finished 2024-04-27T01:15:49.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: osdmap e261: 8 total, 8 up, 8 in 2024-04-27T01:15:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-74307-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:49.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:49 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:49.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-33", "mode": "writeback"}]': finished 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]': finished 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: osdmap e261: 8 total, 8 up, 8 in 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-74307-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:49.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-33", "mode": "writeback"}]': finished 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: from='client.56921 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi162-74307-40"}]': finished 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: osdmap e261: 8 total, 8 up, 8 in 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-74307-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:49.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:50.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:15:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:15:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:15:50.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]': finished 2024-04-27T01:15:50.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-74307-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:50.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:50.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: osdmap e262: 8 total, 8 up, 8 in 2024-04-27T01:15:50.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-74307-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:50.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:50.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: pgmap v351: 228 pgs: 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:50.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:50.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]': finished 2024-04-27T01:15:50.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-74307-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:50.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:50.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: osdmap e262: 8 total, 8 up, 8 in 2024-04-27T01:15:50.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-74307-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:50.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:50.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:50.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: pgmap v351: 228 pgs: 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi162-74325-50"}]': finished 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi162-74307-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: osdmap e262: 8 total, 8 up, 8 in 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-74307-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1194454994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: from='client.56927 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]: dispatch 2024-04-27T01:15:50.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:50.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33"}]: dispatch 2024-04-27T01:15:50.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: pgmap v351: 228 pgs: 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:50.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:15:50.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:15:51.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:51 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:51.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:51 smithi181 ceph-mon[30237]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]': finished 2024-04-27T01:15:51.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:51 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33"}]': finished 2024-04-27T01:15:51.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:51 smithi181 ceph-mon[30237]: osdmap e263: 8 total, 8 up, 8 in 2024-04-27T01:15:51.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:51 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:51.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:51 smithi162 ceph-mon[37680]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]': finished 2024-04-27T01:15:51.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:51 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33"}]': finished 2024-04-27T01:15:51.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:51 smithi162 ceph-mon[37680]: osdmap e263: 8 total, 8 up, 8 in 2024-04-27T01:15:51.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:51 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:15:51.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:51 smithi162 ceph-mon[26714]: from='client.56927 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi162-74325-50"}]': finished 2024-04-27T01:15:51.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:51 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-33"}]': finished 2024-04-27T01:15:51.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:51 smithi162 ceph-mon[26714]: osdmap e263: 8 total, 8 up, 8 in 2024-04-27T01:15:52.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:52 smithi181 ceph-mon[30237]: pgmap v353: 228 pgs: 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:52.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-74307-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-74307-41"}]': finished 2024-04-27T01:15:52.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:52 smithi181 ceph-mon[30237]: osdmap e264: 8 total, 8 up, 8 in 2024-04-27T01:15:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/863658180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:52.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:52 smithi181 ceph-mon[30237]: from='client.56939 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:52.510 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:15:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:15:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:52 smithi162 ceph-mon[37680]: pgmap v353: 228 pgs: 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-74307-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-74307-41"}]': finished 2024-04-27T01:15:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:52 smithi162 ceph-mon[37680]: osdmap e264: 8 total, 8 up, 8 in 2024-04-27T01:15:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/863658180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:52 smithi162 ceph-mon[37680]: from='client.56939 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:52 smithi162 ceph-mon[26714]: pgmap v353: 228 pgs: 228 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:15:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi162-74307-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi162-74307-41"}]': finished 2024-04-27T01:15:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:52 smithi162 ceph-mon[26714]: osdmap e264: 8 total, 8 up, 8 in 2024-04-27T01:15:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/863658180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:52.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:52 smithi162 ceph-mon[26714]: from='client.56939 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:53.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:53 smithi181 ceph-mon[30237]: from='client.56939 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:53.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:53 smithi181 ceph-mon[30237]: osdmap e265: 8 total, 8 up, 8 in 2024-04-27T01:15:53.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:53 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:53.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:53 smithi162 ceph-mon[37680]: from='client.56939 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:53 smithi162 ceph-mon[37680]: osdmap e265: 8 total, 8 up, 8 in 2024-04-27T01:15:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:53 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:53 smithi162 ceph-mon[26714]: from='client.56939 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi162-74325-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:53 smithi162 ceph-mon[26714]: osdmap e265: 8 total, 8 up, 8 in 2024-04-27T01:15:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:53 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:15:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:15:54.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: pgmap v356: 268 pgs: 12 creating+activating, 39 creating+peering, 21 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:54.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:54.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: osdmap e266: 8 total, 8 up, 8 in 2024-04-27T01:15:54.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:54.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:54.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:54 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: pgmap v356: 268 pgs: 12 creating+activating, 39 creating+peering, 21 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: osdmap e266: 8 total, 8 up, 8 in 2024-04-27T01:15:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:54.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:54.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: pgmap v356: 268 pgs: 12 creating+activating, 39 creating+peering, 21 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: osdmap e266: 8 total, 8 up, 8 in 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:54.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:54 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:55.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:55 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-74307-41"}]': finished 2024-04-27T01:15:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:55 smithi181 ceph-mon[30237]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:15:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-74325-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:55 smithi181 ceph-mon[30237]: osdmap e267: 8 total, 8 up, 8 in 2024-04-27T01:15:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:55 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:15:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:55 smithi181 ceph-mon[30237]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-74325-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:55.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-74307-41"}]': finished 2024-04-27T01:15:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-74307-41"}]': finished 2024-04-27T01:15:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[26714]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:15:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-74325-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[26714]: osdmap e267: 8 total, 8 up, 8 in 2024-04-27T01:15:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:15:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[26714]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-74325-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-74307-41"}]': finished 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi162-74307-41"}]': finished 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[37680]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-74325-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[37680]: osdmap e267: 8 total, 8 up, 8 in 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-74307-41"}]: dispatch 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[37680]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-74325-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:55.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4180648413' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi162-74307-41"}]': finished 2024-04-27T01:15:56.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: pgmap v359: 228 pgs: 11 creating+peering, 21 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:56.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]': finished 2024-04-27T01:15:56.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-35", "mode": "writeback"}]: dispatch 2024-04-27T01:15:56.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: osdmap e268: 8 total, 8 up, 8 in 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-35", "mode": "writeback"}]: dispatch 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-74307-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-74307-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-74325-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]': finished 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-35", "mode": "writeback"}]': finished 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-74307-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: osdmap e269: 8 total, 8 up, 8 in 2024-04-27T01:15:56.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-74307-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.761 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:56 smithi181 ceph-mon[30237]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-74307-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: pgmap v359: 228 pgs: 11 creating+peering, 21 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:56.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]': finished 2024-04-27T01:15:56.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-35", "mode": "writeback"}]: dispatch 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: osdmap e268: 8 total, 8 up, 8 in 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-35", "mode": "writeback"}]: dispatch 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-74307-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-74307-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-74325-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]': finished 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-35", "mode": "writeback"}]': finished 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-74307-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:56.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: osdmap e269: 8 total, 8 up, 8 in 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-74307-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[26714]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-74307-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: pgmap v359: 228 pgs: 11 creating+peering, 21 unknown, 196 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]': finished 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-35", "mode": "writeback"}]: dispatch 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: osdmap e268: 8 total, 8 up, 8 in 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-35", "mode": "writeback"}]: dispatch 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-74307-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:56.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-74307-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:15:56.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:15:56.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi162-74325-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]': finished 2024-04-27T01:15:56.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-35", "mode": "writeback"}]': finished 2024-04-27T01:15:56.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi162-74307-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:15:56.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: osdmap e269: 8 total, 8 up, 8 in 2024-04-27T01:15:56.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-74307-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:56.840 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:56 smithi162 ceph-mon[37680]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-74307-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:15:58.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:58 smithi181 ceph-mon[30237]: pgmap v362: 236 pgs: 16 creating+peering, 5 unknown, 215 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:58.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:58 smithi181 ceph-mon[30237]: osdmap e270: 8 total, 8 up, 8 in 2024-04-27T01:15:58.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:58 smithi162 ceph-mon[37680]: pgmap v362: 236 pgs: 16 creating+peering, 5 unknown, 215 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:58.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:58 smithi162 ceph-mon[37680]: osdmap e270: 8 total, 8 up, 8 in 2024-04-27T01:15:58.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:58 smithi162 ceph-mon[26714]: pgmap v362: 236 pgs: 16 creating+peering, 5 unknown, 215 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:15:58.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:58 smithi162 ceph-mon[26714]: osdmap e270: 8 total, 8 up, 8 in 2024-04-27T01:15:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:59 smithi181 ceph-mon[30237]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-74307-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-74307-42"}]': finished 2024-04-27T01:15:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:59 smithi181 ceph-mon[30237]: osdmap e271: 8 total, 8 up, 8 in 2024-04-27T01:15:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:59 smithi181 ceph-mon[30237]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:59 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:15:59 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:59.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[26714]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-74307-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-74307-42"}]': finished 2024-04-27T01:15:59.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[26714]: osdmap e271: 8 total, 8 up, 8 in 2024-04-27T01:15:59.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:59.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[26714]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:59.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:59.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:59.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:15:59.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[37680]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi162-74307-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi162-74307-42"}]': finished 2024-04-27T01:15:59.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[37680]: osdmap e271: 8 total, 8 up, 8 in 2024-04-27T01:15:59.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:59.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[37680]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:15:59.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:59.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:15:59.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:00.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:15:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:15:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:16:00.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:00 smithi181 ceph-mon[30237]: pgmap v365: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:16:00.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:00 smithi181 ceph-mon[30237]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]': finished 2024-04-27T01:16:00.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:00 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:00.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:16:00.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:00 smithi181 ceph-mon[30237]: osdmap e272: 8 total, 8 up, 8 in 2024-04-27T01:16:00.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:00 smithi181 ceph-mon[30237]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:16:00.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:00.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:00 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:00.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[26714]: pgmap v365: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:16:00.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[26714]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]': finished 2024-04-27T01:16:00.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:00.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:16:00.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[26714]: osdmap e272: 8 total, 8 up, 8 in 2024-04-27T01:16:00.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[26714]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:16:00.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:00.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:00.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[37680]: pgmap v365: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:16:00.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[37680]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi162-74325-52"}]': finished 2024-04-27T01:16:00.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:00.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1561059975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:16:00.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[37680]: osdmap e272: 8 total, 8 up, 8 in 2024-04-27T01:16:00.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[37680]: from='client.49195 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]: dispatch 2024-04-27T01:16:00.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:00.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:00 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:01.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:01 smithi181 ceph-mon[30237]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]': finished 2024-04-27T01:16:01.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:01 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]': finished 2024-04-27T01:16:01.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:01 smithi181 ceph-mon[30237]: osdmap e273: 8 total, 8 up, 8 in 2024-04-27T01:16:01.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:01.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:01 smithi181 ceph-mon[30237]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:01.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:01.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:01.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[37680]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]': finished 2024-04-27T01:16:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]': finished 2024-04-27T01:16:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[37680]: osdmap e273: 8 total, 8 up, 8 in 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[37680]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[26714]: from='client.49195 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi162-74325-52"}]': finished 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-35"}]': finished 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[26714]: osdmap e273: 8 total, 8 up, 8 in 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[26714]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:01.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:01.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:01.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:02.249 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7027 ms) 2024-04-27T01:16:02.249 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-04-27T01:16:02.249 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6931 ms) 2024-04-27T01:16:02.249 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-04-27T01:16:02.249 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7034 ms) 2024-04-27T01:16:02.249 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-04-27T01:16:02.249 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6987 ms) 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7073 ms) 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7052 ms) 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7034 ms) 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6838 ms) 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7033 ms) 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7029 ms) 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6941 ms) 2024-04-27T01:16:02.250 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7030 ms) 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6972 ms) 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7042 ms) 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7273 ms) 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6802 ms) 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112098 ms total) 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [----------] Global test environment tear-down 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (209589 ms total) 2024-04-27T01:16:02.251 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio: [ PASSED ] 42 tests. 2024-04-27T01:16:02.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:16:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:16:02.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:02 smithi181 ceph-mon[30237]: pgmap v368: 228 pgs: 228 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-27T01:16:02.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:02 smithi181 ceph-mon[30237]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]': finished 2024-04-27T01:16:02.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:02.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:02 smithi181 ceph-mon[30237]: osdmap e274: 8 total, 8 up, 8 in 2024-04-27T01:16:02.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-74325-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:02.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:02.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:02 smithi181 ceph-mon[30237]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:02.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:02.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:02.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[26714]: pgmap v368: 228 pgs: 228 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-27T01:16:02.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[26714]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]': finished 2024-04-27T01:16:02.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:02.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[26714]: osdmap e274: 8 total, 8 up, 8 in 2024-04-27T01:16:02.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-74325-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:02.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:02.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[26714]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:02.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:02.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[37680]: pgmap v368: 228 pgs: 228 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-27T01:16:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[37680]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi162-74307-42"}]': finished 2024-04-27T01:16:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[37680]: osdmap e274: 8 total, 8 up, 8 in 2024-04-27T01:16:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-74325-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3252460361' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[37680]: from='client.56951 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]: dispatch 2024-04-27T01:16:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:02.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:03.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:03 smithi181 ceph-mon[30237]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]': finished 2024-04-27T01:16:03.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:03.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:03.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:03.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:03 smithi181 ceph-mon[30237]: osdmap e275: 8 total, 8 up, 8 in 2024-04-27T01:16:03.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:03.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[26714]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]': finished 2024-04-27T01:16:03.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:03.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:03.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:03.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[26714]: osdmap e275: 8 total, 8 up, 8 in 2024-04-27T01:16:03.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:03.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[37680]: from='client.56951 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi162-74307-42"}]': finished 2024-04-27T01:16:03.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:03.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:03.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:03.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[37680]: osdmap e275: 8 total, 8 up, 8 in 2024-04-27T01:16:03.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35"}]: dispatch 2024-04-27T01:16:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:04 smithi181 ceph-mon[30237]: pgmap v371: 228 pgs: 228 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:16:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:04 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:16:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-74325-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-74325-53"}]': finished 2024-04-27T01:16:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35"}]': finished 2024-04-27T01:16:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:04 smithi181 ceph-mon[30237]: osdmap e276: 8 total, 8 up, 8 in 2024-04-27T01:16:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:04 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:04 smithi181 ceph-mon[30237]: osdmap e277: 8 total, 8 up, 8 in 2024-04-27T01:16:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[26714]: pgmap v371: 228 pgs: 228 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:16:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:16:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-74325-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-74325-53"}]': finished 2024-04-27T01:16:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35"}]': finished 2024-04-27T01:16:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[26714]: osdmap e276: 8 total, 8 up, 8 in 2024-04-27T01:16:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[26714]: osdmap e277: 8 total, 8 up, 8 in 2024-04-27T01:16:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[37680]: pgmap v371: 228 pgs: 228 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:16:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:16:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi162-74325-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi162-74325-53"}]': finished 2024-04-27T01:16:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-35"}]': finished 2024-04-27T01:16:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[37680]: osdmap e276: 8 total, 8 up, 8 in 2024-04-27T01:16:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:04.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:04 smithi162 ceph-mon[37680]: osdmap e277: 8 total, 8 up, 8 in 2024-04-27T01:16:06.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:06 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:06.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:06 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:06.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:06 smithi181 ceph-mon[30237]: osdmap e278: 8 total, 8 up, 8 in 2024-04-27T01:16:06.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:06.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:06.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:06 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:06.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:06.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:06.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[26714]: osdmap e278: 8 total, 8 up, 8 in 2024-04-27T01:16:06.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:06.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:06.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:06.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:06.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:06.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[37680]: osdmap e278: 8 total, 8 up, 8 in 2024-04-27T01:16:06.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:06.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:06.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:06 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:07.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:07 smithi181 ceph-mon[30237]: pgmap v375: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 518 B/s wr, 2 op/s 2024-04-27T01:16:07.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53"}]': finished 2024-04-27T01:16:07.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:07 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:07.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:07 smithi181 ceph-mon[30237]: osdmap e279: 8 total, 8 up, 8 in 2024-04-27T01:16:07.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:07.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:07.260 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:07 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:07.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[37680]: pgmap v375: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 518 B/s wr, 2 op/s 2024-04-27T01:16:07.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53"}]': finished 2024-04-27T01:16:07.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:07.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[37680]: osdmap e279: 8 total, 8 up, 8 in 2024-04-27T01:16:07.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:07.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:07.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:07.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[26714]: pgmap v375: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 518 B/s wr, 2 op/s 2024-04-27T01:16:07.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi162-74325-53"}]': finished 2024-04-27T01:16:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[26714]: osdmap e279: 8 total, 8 up, 8 in 2024-04-27T01:16:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-74325-53"}]: dispatch 2024-04-27T01:16:07.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:07 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-74325-53"}]': finished 2024-04-27T01:16:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:08 smithi181 ceph-mon[30237]: osdmap e280: 8 total, 8 up, 8 in 2024-04-27T01:16:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-74325-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:08.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:08 smithi181 ceph-mon[30237]: pgmap v378: 228 pgs: 12 unknown, 216 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-04-27T01:16:08.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-74325-53"}]': finished 2024-04-27T01:16:08.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:08.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[26714]: osdmap e280: 8 total, 8 up, 8 in 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-74325-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[26714]: pgmap v378: 228 pgs: 12 unknown, 216 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1601501397' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi162-74325-53"}]': finished 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[37680]: osdmap e280: 8 total, 8 up, 8 in 2024-04-27T01:16:08.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:08.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:08.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:08.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-74325-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:08.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:08 smithi162 ceph-mon[37680]: pgmap v378: 228 pgs: 12 unknown, 216 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-04-27T01:16:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-37"}]': finished 2024-04-27T01:16:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-74325-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-37", "mode": "writeback"}]: dispatch 2024-04-27T01:16:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:09 smithi181 ceph-mon[30237]: osdmap e281: 8 total, 8 up, 8 in 2024-04-27T01:16:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-74325-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-37", "mode": "writeback"}]: dispatch 2024-04-27T01:16:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-37"}]': finished 2024-04-27T01:16:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-74325-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-37", "mode": "writeback"}]: dispatch 2024-04-27T01:16:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[26714]: osdmap e281: 8 total, 8 up, 8 in 2024-04-27T01:16:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-74325-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:09.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-37", "mode": "writeback"}]: dispatch 2024-04-27T01:16:09.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-37"}]': finished 2024-04-27T01:16:09.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi162-74325-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:09.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-37", "mode": "writeback"}]: dispatch 2024-04-27T01:16:09.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[37680]: osdmap e281: 8 total, 8 up, 8 in 2024-04-27T01:16:09.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-74325-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:09.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-37", "mode": "writeback"}]: dispatch 2024-04-27T01:16:10.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:16:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:16:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:16:10.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:10 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:16:10.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-37", "mode": "writeback"}]': finished 2024-04-27T01:16:10.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:10 smithi181 ceph-mon[30237]: pgmap v381: 228 pgs: 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:10.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:10.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:10 smithi181 ceph-mon[30237]: osdmap e282: 8 total, 8 up, 8 in 2024-04-27T01:16:10.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:10.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:16:10.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-37", "mode": "writeback"}]': finished 2024-04-27T01:16:10.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[37680]: pgmap v381: 228 pgs: 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:10.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:10.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[37680]: osdmap e282: 8 total, 8 up, 8 in 2024-04-27T01:16:10.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:10.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:16:10.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-37", "mode": "writeback"}]': finished 2024-04-27T01:16:10.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[26714]: pgmap v381: 228 pgs: 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:10.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:10.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[26714]: osdmap e282: 8 total, 8 up, 8 in 2024-04-27T01:16:10.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:11.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-74325-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-74325-54"}]': finished 2024-04-27T01:16:11.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:11 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:11.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:11.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:11 smithi181 ceph-mon[30237]: osdmap e283: 8 total, 8 up, 8 in 2024-04-27T01:16:11.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:11 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:11.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-74325-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-74325-54"}]': finished 2024-04-27T01:16:11.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:11 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:11.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:11.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:11 smithi162 ceph-mon[37680]: osdmap e283: 8 total, 8 up, 8 in 2024-04-27T01:16:11.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:11 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:11.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi162-74325-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi162-74325-54"}]': finished 2024-04-27T01:16:11.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:11 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:11.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:11.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:11 smithi162 ceph-mon[26714]: osdmap e283: 8 total, 8 up, 8 in 2024-04-27T01:16:11.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:11 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37"}]: dispatch 2024-04-27T01:16:12.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:12 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:16:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37"}]': finished 2024-04-27T01:16:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:12 smithi162 ceph-mon[37680]: osdmap e284: 8 total, 8 up, 8 in 2024-04-27T01:16:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:12 smithi162 ceph-mon[37680]: pgmap v384: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:12 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:12.489 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:12 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:16:12.489 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37"}]': finished 2024-04-27T01:16:12.489 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:12 smithi162 ceph-mon[26714]: osdmap e284: 8 total, 8 up, 8 in 2024-04-27T01:16:12.489 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:12 smithi162 ceph-mon[26714]: pgmap v384: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:12.489 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:12 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:12.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:16:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:16:12.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:12 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:16:12.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-37"}]': finished 2024-04-27T01:16:12.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:12 smithi181 ceph-mon[30237]: osdmap e284: 8 total, 8 up, 8 in 2024-04-27T01:16:12.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:12 smithi181 ceph-mon[30237]: pgmap v384: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:12.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:12 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:16:13.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:13 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:13.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:13 smithi181 ceph-mon[30237]: osdmap e285: 8 total, 8 up, 8 in 2024-04-27T01:16:13.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:13.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:13.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:13 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:13.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:13 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:13.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:13 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:16:13.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:13 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:16:13.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:13 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[37680]: osdmap e285: 8 total, 8 up, 8 in 2024-04-27T01:16:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:16:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:16:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:13.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:13.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[26714]: osdmap e285: 8 total, 8 up, 8 in 2024-04-27T01:16:13.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:13.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:13.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:13.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:13.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:16:13.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:16:13.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:13 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:16:14.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-74325-54"}]': finished 2024-04-27T01:16:14.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:14 smithi181 ceph-mon[30237]: osdmap e286: 8 total, 8 up, 8 in 2024-04-27T01:16:14.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:14.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:14.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:14 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:14.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:14 smithi181 ceph-mon[30237]: pgmap v387: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:16:14.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-74325-54"}]': finished 2024-04-27T01:16:14.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[37680]: osdmap e286: 8 total, 8 up, 8 in 2024-04-27T01:16:14.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:14.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:14.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:14.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[37680]: pgmap v387: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:16:14.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi162-74325-54"}]': finished 2024-04-27T01:16:14.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[26714]: osdmap e286: 8 total, 8 up, 8 in 2024-04-27T01:16:14.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-74325-54"}]: dispatch 2024-04-27T01:16:14.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:14.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:14.590 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:14 smithi162 ceph-mon[26714]: pgmap v387: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:16:15.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-74325-54"}]': finished 2024-04-27T01:16:15.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:15.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: osdmap e287: 8 total, 8 up, 8 in 2024-04-27T01:16:15.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:15.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:15.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-74325-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:15.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:15 smithi181 ceph-mon[30237]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-74325-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-74325-54"}]': finished 2024-04-27T01:16:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: osdmap e287: 8 total, 8 up, 8 in 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-74325-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[37680]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-74325-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1290268590' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi162-74325-54"}]': finished 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: osdmap e287: 8 total, 8 up, 8 in 2024-04-27T01:16:15.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:15.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:15.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:15.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-74325-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:15.589 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:15 smithi162 ceph-mon[26714]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-74325-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:16.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:16 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:16.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:16 smithi181 ceph-mon[30237]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-74325-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:16.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:16 smithi181 ceph-mon[30237]: osdmap e288: 8 total, 8 up, 8 in 2024-04-27T01:16:16.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-74325-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:16.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:16.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:16 smithi181 ceph-mon[30237]: pgmap v390: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:16:16.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:16 smithi181 ceph-mon[30237]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-74325-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:16.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:16 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:16.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:16.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[37680]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-74325-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:16.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[37680]: osdmap e288: 8 total, 8 up, 8 in 2024-04-27T01:16:16.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-74325-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:16.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:16.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[37680]: pgmap v390: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:16:16.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[37680]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-74325-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:16.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:16.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:16.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[26714]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi162-74325-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:16.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[26714]: osdmap e288: 8 total, 8 up, 8 in 2024-04-27T01:16:16.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-74325-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:16.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:16.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[26714]: pgmap v390: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:16:16.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[26714]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-74325-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:16.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:16 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:17.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:17 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-39"}]': finished 2024-04-27T01:16:17.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-39", "mode": "writeback"}]: dispatch 2024-04-27T01:16:17.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:17 smithi181 ceph-mon[30237]: osdmap e289: 8 total, 8 up, 8 in 2024-04-27T01:16:17.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:17 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-39", "mode": "writeback"}]: dispatch 2024-04-27T01:16:17.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:17 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-39"}]': finished 2024-04-27T01:16:17.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-39", "mode": "writeback"}]: dispatch 2024-04-27T01:16:17.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:17 smithi162 ceph-mon[26714]: osdmap e289: 8 total, 8 up, 8 in 2024-04-27T01:16:17.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:17 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-39", "mode": "writeback"}]: dispatch 2024-04-27T01:16:17.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:17 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-39"}]': finished 2024-04-27T01:16:17.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-39", "mode": "writeback"}]: dispatch 2024-04-27T01:16:17.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:17 smithi162 ceph-mon[37680]: osdmap e289: 8 total, 8 up, 8 in 2024-04-27T01:16:17.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:17 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-39", "mode": "writeback"}]: dispatch 2024-04-27T01:16:18.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:18 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:16:18.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:18 smithi181 ceph-mon[30237]: pgmap v392: 228 pgs: 13 unknown, 215 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:18.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:18 smithi181 ceph-mon[30237]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-74325-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-74325-55"}]': finished 2024-04-27T01:16:18.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:18 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-39", "mode": "writeback"}]': finished 2024-04-27T01:16:18.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:18 smithi181 ceph-mon[30237]: osdmap e290: 8 total, 8 up, 8 in 2024-04-27T01:16:18.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:18.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:18 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:18.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:16:18.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[37680]: pgmap v392: 228 pgs: 13 unknown, 215 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:18.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[37680]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-74325-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-74325-55"}]': finished 2024-04-27T01:16:18.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-39", "mode": "writeback"}]': finished 2024-04-27T01:16:18.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[37680]: osdmap e290: 8 total, 8 up, 8 in 2024-04-27T01:16:18.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:18.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:18.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:16:18.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[26714]: pgmap v392: 228 pgs: 13 unknown, 215 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:18.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[26714]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi162-74325-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi162-74325-55"}]': finished 2024-04-27T01:16:18.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-39", "mode": "writeback"}]': finished 2024-04-27T01:16:18.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[26714]: osdmap e290: 8 total, 8 up, 8 in 2024-04-27T01:16:18.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:18.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:18 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:19.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:19 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:19.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:19 smithi181 ceph-mon[30237]: osdmap e291: 8 total, 8 up, 8 in 2024-04-27T01:16:19.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:19.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:19 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:19.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:19 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:19.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:19.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-mon[37680]: osdmap e291: 8 total, 8 up, 8 in 2024-04-27T01:16:19.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:19.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:19.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:19.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:16:19.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-mon[26714]: osdmap e291: 8 total, 8 up, 8 in 2024-04-27T01:16:19.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:19.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39"}]: dispatch 2024-04-27T01:16:19.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:20.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:16:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:16:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:16:20.212 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-04-27T01:16:20.212 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-04-27T01:16:20.212 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-04-27T01:16:20.212 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: seed 74551 2024-04-27T01:16:20.212 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-04-27T01:16:20.212 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (316 ms) 2024-04-27T01:16:20.212 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (14669 ms) 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (135 ms) 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (15120 ms total) 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (10010 ms) 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (10676 ms) 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10020 ms) 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: my_snaps [3] 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: my_snaps [4,3] 2024-04-27T01:16:20.213 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: my_snaps [5,4,3] 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: promoting some heads 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: promoting from clones for snap 6 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: promoting from clones for snap 5 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: promoting from clones for snap 4 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: promoting from clones for snap 3 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: waiting for scrubs... 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: done waiting 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46236 ms) 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9791 ms) 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7982 ms) 2024-04-27T01:16:20.214 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8085 ms) 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8058 ms) 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10036 ms) 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8859 ms) 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10275 ms) 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12731 ms) 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8224 ms) 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-04-27T01:16:20.215 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7820 ms) 2024-04-27T01:16:20.216 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-04-27T01:16:20.216 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12926 ms) 2024-04-27T01:16:20.216 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-04-27T01:16:20.216 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8040 ms) 2024-04-27T01:16:20.216 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-04-27T01:16:20.216 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8046 ms) 2024-04-27T01:16:20.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:20 smithi181 ceph-mon[30237]: pgmap v395: 236 pgs: 4 creating+peering, 4 unknown, 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:20.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:20 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:16:20.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:20 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39"}]': finished 2024-04-27T01:16:20.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:20 smithi181 ceph-mon[30237]: osdmap e292: 8 total, 8 up, 8 in 2024-04-27T01:16:20.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:20.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:20 smithi181 ceph-mon[30237]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:20.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:20.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:20 smithi181 ceph-mon[30237]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]': finished 2024-04-27T01:16:20.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:20 smithi181 ceph-mon[30237]: osdmap e293: 8 total, 8 up, 8 in 2024-04-27T01:16:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[37680]: pgmap v395: 236 pgs: 4 creating+peering, 4 unknown, 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:16:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39"}]': finished 2024-04-27T01:16:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[37680]: osdmap e292: 8 total, 8 up, 8 in 2024-04-27T01:16:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[37680]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[37680]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]': finished 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[37680]: osdmap e293: 8 total, 8 up, 8 in 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[26714]: pgmap v395: 236 pgs: 4 creating+peering, 4 unknown, 228 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-39"}]': finished 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[26714]: osdmap e292: 8 total, 8 up, 8 in 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[26714]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[26714]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi162-74325-55"}]': finished 2024-04-27T01:16:20.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:20 smithi162 ceph-mon[26714]: osdmap e293: 8 total, 8 up, 8 in 2024-04-27T01:16:21.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:21.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:21 smithi181 ceph-mon[30237]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:21.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:21 smithi181 ceph-mon[30237]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]': finished 2024-04-27T01:16:21.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:21.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:21 smithi162 ceph-mon[26714]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:21.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:21 smithi162 ceph-mon[26714]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]': finished 2024-04-27T01:16:21.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2064261815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:21.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:21 smithi162 ceph-mon[37680]: from='client.56969 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]: dispatch 2024-04-27T01:16:21.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:21 smithi162 ceph-mon[37680]: from='client.56969 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi162-74325-55"}]': finished 2024-04-27T01:16:22.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:16:22.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: pgmap v398: 196 pgs: 196 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:22.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: osdmap e294: 8 total, 8 up, 8 in 2024-04-27T01:16:22.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:22.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:22.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:22.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:22.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-74325-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:22.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:22.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-74325-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:22.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: osdmap e295: 8 total, 8 up, 8 in 2024-04-27T01:16:22.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-74325-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:22.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:22.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:22 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:22.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: pgmap v398: 196 pgs: 196 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:22.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: osdmap e294: 8 total, 8 up, 8 in 2024-04-27T01:16:22.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:22.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:22.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:22.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:22.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-74325-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:22.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-74325-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: osdmap e295: 8 total, 8 up, 8 in 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-74325-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: pgmap v398: 196 pgs: 196 active+clean; 4.6 MiB data, 586 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: osdmap e294: 8 total, 8 up, 8 in 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:22.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-74325-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:22.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:22.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi162-74325-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:22.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: osdmap e295: 8 total, 8 up, 8 in 2024-04-27T01:16:22.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-74325-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:22.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:22.589 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:22 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:23.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:23.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:23 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:23.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:23.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:23 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:23.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:23.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:23 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:24.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:24 smithi181 ceph-mon[30237]: pgmap v401: 228 pgs: 4 creating+activating, 28 creating+peering, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-04-27T01:16:24.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:16:24.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:24 smithi181 ceph-mon[30237]: osdmap e296: 8 total, 8 up, 8 in 2024-04-27T01:16:24.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:24 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:16:24.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-74325-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-74325-56"}]': finished 2024-04-27T01:16:24.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:24 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:16:24.762 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:24.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:24 smithi181 ceph-mon[30237]: osdmap e297: 8 total, 8 up, 8 in 2024-04-27T01:16:24.763 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:24 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[37680]: pgmap v401: 228 pgs: 4 creating+activating, 28 creating+peering, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-04-27T01:16:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:16:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[37680]: osdmap e296: 8 total, 8 up, 8 in 2024-04-27T01:16:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:16:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-74325-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-74325-56"}]': finished 2024-04-27T01:16:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:16:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[37680]: osdmap e297: 8 total, 8 up, 8 in 2024-04-27T01:16:24.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[26714]: pgmap v401: 228 pgs: 4 creating+activating, 28 creating+peering, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-04-27T01:16:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:16:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[26714]: osdmap e296: 8 total, 8 up, 8 in 2024-04-27T01:16:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:16:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi162-74325-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi162-74325-56"}]': finished 2024-04-27T01:16:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:16:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[26714]: osdmap e297: 8 total, 8 up, 8 in 2024-04-27T01:16:24.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:24 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:26.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:26 smithi181 ceph-mon[30237]: pgmap v404: 236 pgs: 8 unknown, 4 creating+activating, 28 creating+peering, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-27T01:16:26.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:26 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:16:26.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:26 smithi181 ceph-mon[30237]: osdmap e298: 8 total, 8 up, 8 in 2024-04-27T01:16:26.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:16:26.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:26 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:16:26.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:26 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:26.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[37680]: pgmap v404: 236 pgs: 8 unknown, 4 creating+activating, 28 creating+peering, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-27T01:16:26.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:16:26.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[37680]: osdmap e298: 8 total, 8 up, 8 in 2024-04-27T01:16:26.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:16:26.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:16:26.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:26.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[26714]: pgmap v404: 236 pgs: 8 unknown, 4 creating+activating, 28 creating+peering, 196 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-27T01:16:26.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:16:26.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[26714]: osdmap e298: 8 total, 8 up, 8 in 2024-04-27T01:16:26.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:16:26.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:16:26.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:26 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:27.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:27 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:16:27.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:27 smithi181 ceph-mon[30237]: osdmap e299: 8 total, 8 up, 8 in 2024-04-27T01:16:27.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:27.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-74325-56"}]': finished 2024-04-27T01:16:27.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:27 smithi181 ceph-mon[30237]: osdmap e300: 8 total, 8 up, 8 in 2024-04-27T01:16:27.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:27 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:16:27.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:27 smithi162 ceph-mon[26714]: osdmap e299: 8 total, 8 up, 8 in 2024-04-27T01:16:27.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:27.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-74325-56"}]': finished 2024-04-27T01:16:27.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:27 smithi162 ceph-mon[26714]: osdmap e300: 8 total, 8 up, 8 in 2024-04-27T01:16:27.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:27 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:16:27.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:27 smithi162 ceph-mon[37680]: osdmap e299: 8 total, 8 up, 8 in 2024-04-27T01:16:27.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:27.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi162-74325-56"}]': finished 2024-04-27T01:16:27.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:27 smithi162 ceph-mon[37680]: osdmap e300: 8 total, 8 up, 8 in 2024-04-27T01:16:28.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:28 smithi181 ceph-mon[30237]: pgmap v407: 228 pgs: 4 creating+activating, 10 creating+peering, 214 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:28.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:28.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:28.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:28.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41"}]: dispatch 2024-04-27T01:16:28.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41"}]: dispatch 2024-04-27T01:16:28.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-74325-56"}]': finished 2024-04-27T01:16:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41"}]': finished 2024-04-27T01:16:28.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:28 smithi181 ceph-mon[30237]: osdmap e301: 8 total, 8 up, 8 in 2024-04-27T01:16:28.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[37680]: pgmap v407: 228 pgs: 4 creating+activating, 10 creating+peering, 214 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:28.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:28.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:28.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:28.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41"}]: dispatch 2024-04-27T01:16:28.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41"}]: dispatch 2024-04-27T01:16:28.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-74325-56"}]': finished 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41"}]': finished 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[37680]: osdmap e301: 8 total, 8 up, 8 in 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[26714]: pgmap v407: 228 pgs: 4 creating+activating, 10 creating+peering, 214 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-74325-56"}]: dispatch 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41"}]: dispatch 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41"}]: dispatch 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/206500257' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi162-74325-56"}]': finished 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-41"}]': finished 2024-04-27T01:16:28.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:28 smithi162 ceph-mon[26714]: osdmap e301: 8 total, 8 up, 8 in 2024-04-27T01:16:29.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:29.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:29.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-74325-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:29.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:29.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:29.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-74325-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:29.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:29.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:29.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-74325-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:30.087 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:16:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:16:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:16:30.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:30 smithi181 ceph-mon[30237]: pgmap v410: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:30.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-74325-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:30.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:30 smithi181 ceph-mon[30237]: osdmap e302: 8 total, 8 up, 8 in 2024-04-27T01:16:30.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-74325-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:30.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:30 smithi162 ceph-mon[37680]: pgmap v410: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:30.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-74325-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:30.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:30 smithi162 ceph-mon[37680]: osdmap e302: 8 total, 8 up, 8 in 2024-04-27T01:16:30.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-74325-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:30.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:30 smithi162 ceph-mon[26714]: pgmap v410: 228 pgs: 228 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:30.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi162-74325-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:30 smithi162 ceph-mon[26714]: osdmap e302: 8 total, 8 up, 8 in 2024-04-27T01:16:30.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-74325-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:31.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:31 smithi181 ceph-mon[30237]: osdmap e303: 8 total, 8 up, 8 in 2024-04-27T01:16:31.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:31.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:31 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:31.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-74325-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-74325-57"}]': finished 2024-04-27T01:16:31.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:31 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:31.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:31 smithi181 ceph-mon[30237]: osdmap e304: 8 total, 8 up, 8 in 2024-04-27T01:16:31.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-74551-6","var": "pg_num","format": "json"}]: dispatch 2024-04-27T01:16:31.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:31.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:31 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:31.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[37680]: osdmap e303: 8 total, 8 up, 8 in 2024-04-27T01:16:31.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:31.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:31.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-74325-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-74325-57"}]': finished 2024-04-27T01:16:31.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:31.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[37680]: osdmap e304: 8 total, 8 up, 8 in 2024-04-27T01:16:31.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-74551-6","var": "pg_num","format": "json"}]: dispatch 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[26714]: osdmap e303: 8 total, 8 up, 8 in 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi162-74325-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi162-74325-57"}]': finished 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[26714]: osdmap e304: 8 total, 8 up, 8 in 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-74551-6","var": "pg_num","format": "json"}]: dispatch 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:31.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:31 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:32.660 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:16:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:16:32.661 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:32 smithi181 ceph-mon[30237]: pgmap v413: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:32.661 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:32 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:32.661 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:32 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:32.661 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:32 smithi181 ceph-mon[30237]: osdmap e305: 8 total, 8 up, 8 in 2024-04-27T01:16:32.661 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-27T01:16:32.661 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:32 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-27T01:16:32.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[37680]: pgmap v413: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:32.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:32.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:32.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[37680]: osdmap e305: 8 total, 8 up, 8 in 2024-04-27T01:16:32.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-27T01:16:32.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-27T01:16:32.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[26714]: pgmap v413: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:32.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:32.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:32.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[26714]: osdmap e305: 8 total, 8 up, 8 in 2024-04-27T01:16:32.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-27T01:16:32.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:32 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-27T01:16:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:33 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_count","val": "8"}]': finished 2024-04-27T01:16:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:33 smithi181 ceph-mon[30237]: osdmap e306: 8 total, 8 up, 8 in 2024-04-27T01:16:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:33 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_count","val": "8"}]': finished 2024-04-27T01:16:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[37680]: osdmap e306: 8 total, 8 up, 8 in 2024-04-27T01:16:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:33.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:33.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_count","val": "8"}]': finished 2024-04-27T01:16:33.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[26714]: osdmap e306: 8 total, 8 up, 8 in 2024-04-27T01:16:33.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:33.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:16:33.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:34 smithi181 ceph-mon[30237]: pgmap v416: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:16:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-74325-57"}]': finished 2024-04-27T01:16:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:34 smithi181 ceph-mon[30237]: osdmap e307: 8 total, 8 up, 8 in 2024-04-27T01:16:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-27T01:16:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-27T01:16:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[37680]: pgmap v416: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:16:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-74325-57"}]': finished 2024-04-27T01:16:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[37680]: osdmap e307: 8 total, 8 up, 8 in 2024-04-27T01:16:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-27T01:16:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-27T01:16:34.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[26714]: pgmap v416: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:34.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:16:34.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi162-74325-57"}]': finished 2024-04-27T01:16:34.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[26714]: osdmap e307: 8 total, 8 up, 8 in 2024-04-27T01:16:34.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-27T01:16:34.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-74325-57"}]: dispatch 2024-04-27T01:16:34.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-27T01:16:35.484 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: ok, hit_set contains 275:602f83fe:::foo:head 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9032 ms) 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg_num = 32 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 0 ls 1714180596,0 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 1 ls 1714180596,0 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 2 ls 1714180596,0 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 3 ls 1714180596,0 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 4 ls 1714180596,0 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 5 ls 1714180596,0 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 6 ls 1714180596,0 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 7 ls 1714180596,0 2024-04-27T01:16:35.485 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 8 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 9 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 10 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 11 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 12 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 13 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 14 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 15 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 16 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 17 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 18 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 19 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 20 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 21 ls 1714180596,0 2024-04-27T01:16:35.486 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 22 ls 1714180596,0 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 23 ls 1714180596,0 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 24 ls 1714180596,0 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 25 ls 1714180596,0 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 26 ls 1714180596,0 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 27 ls 1714180596,0 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 28 ls 1714180596,0 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 29 ls 1714180596,0 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 30 ls 1714180596,0 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg 31 ls 1714180596,0 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: pg_num = 32 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:6cac518f:::0:head 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:02547ec2:::1:head 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:f905c69b:::2:head 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:cfc208b3:::3:head 2024-04-27T01:16:35.487 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d83876eb:::4:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:b29083e3:::5:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c4fdafeb:::6:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:5c6b0b28:::7:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:bd63b0f1:::8:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:e960b815:::9:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:52ea6a34:::10:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:89d3ae78:::11:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:de5d7c5f:::12:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:566253c9:::13:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:62a1935d:::14:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:863748b0:::15:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:3958e169:::16:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:4d4dabf9:::17:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:8391935d:::18:head 2024-04-27T01:16:35.488 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:28883081:::19:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:69259c59:::20:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:4bdb80b7:::21:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:a11c5d71:::22:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:271af37b:::23:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:95b121be:::24:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:58d1031b:::25:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:0a050783:::26:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c709704c:::27:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:cbe56eaf:::28:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:86b4b162:::29:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:70d89383:::30:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:dd450c7c:::31:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:6d5729b1:::32:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c388f3fb:::33:head 2024-04-27T01:16:35.489 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:56cfea31:::34:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:9dbc1bf7:::35:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:40b74ccd:::36:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:4d5aaf42:::37:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:920f362c:::38:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:6cc53222:::39:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:9cad833f:::40:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:1ea84d41:::41:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c4480ef6:::42:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:a694361e:::43:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d1bd33e9:::44:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:ddc2cd5d:::45:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:2b782207:::46:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:7b187fca:::47:head 2024-04-27T01:16:35.490 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:90ecdf6f:::48:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:a5ed95fe:::49:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:ea0eaa55:::50:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:f33ef17b:::51:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:a0d1b2f6:::52:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:60c5229e:::53:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:edcbc575:::54:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:102cf253:::55:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:efb7fb0b:::56:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:50d0a326:::57:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d4dc5daf:::58:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:3a130462:::59:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:ec87ed71:::60:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d5bc9454:::61:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:3ddfe313:::62:head 2024-04-27T01:16:35.491 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:7c2816b9:::63:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:47e00e4d:::64:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c6410c18:::65:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:b48ed237:::66:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:cd63ad31:::67:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:b179e92b:::68:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:0d9f741a:::69:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:6d3352ae:::70:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c6d5c19e:::71:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:bc4729c3:::72:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:77e930b9:::73:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:0abeecfd:::74:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:b7c37e15:::75:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:b6378398:::76:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:02bd68de:::77:head 2024-04-27T01:16:35.492 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:cc795d2d:::78:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:630d4fea:::79:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:e0d29ef5:::80:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:fd6f13d2:::81:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:606461d5:::82:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:eadbdc43:::83:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:8761d0bb:::84:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:9ef0186f:::85:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:e0d41294:::86:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:961de695:::87:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:1423148f:::88:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:633a8fa2:::89:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:a8653809:::90:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:3dac8b33:::91:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:35aad435:::92:head 2024-04-27T01:16:35.493 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:f6dcc343:::93:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:dbbdad87:::94:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:1cb48ce0:::95:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:03cd461c:::96:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:17a4ea99:::97:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:9993c9a7:::98:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:6394211c:::99:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:94c7ae57:::100:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:6fdee5bb:::101:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:9a477fd1:::102:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:eb850916:::103:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:affc56b9:::104:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:b42dc814:::105:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:f319f8f0:::106:head 2024-04-27T01:16:35.494 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:9a40b9de:::107:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:8b524f28:::108:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:e3de589f:::109:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:90f90a5b:::110:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:a7b4f1d7:::111:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:af51766e:::112:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:b6f90bd1:::113:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:e0261208:::114:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c9569ef7:::115:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:61bebe50:::116:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:fe93412b:::117:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d3d38bee:::118:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:3100ba0c:::119:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d0560ada:::120:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:f0ea8b35:::121:head 2024-04-27T01:16:35.495 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:766f231a:::122:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:a07a2582:::123:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:bd7c6b3a:::124:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:fb2ddaff:::125:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:4408e1fe:::126:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:ee1df7a7:::127:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c3002909:::128:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:4f48ffa9:::129:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:edf38733:::130:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c08425c0:::131:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:5f902d98:::132:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:41ea2c93:::133:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:813cee13:::134:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:0131818d:::135:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:26ba5a85:::136:head 2024-04-27T01:16:35.496 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:381b8a5a:::137:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:28797e47:::138:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:bfca7f22:::139:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:36807075:::140:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:80b03975:::141:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:5c15709b:::142:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:f39ea15e:::143:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:ea992956:::144:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:48887b1c:::145:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:9f24a9dd:::146:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:987f100b:::147:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d2dd3581:::148:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:7fed1808:::149:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c80b70e9:::150:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:85ed90f9:::151:head 2024-04-27T01:16:35.497 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:36428b24:::152:head 2024-04-27T01:16:35.498 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d044c34a:::153:head 2024-04-27T01:16:35.498 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:7c18bf58:::154:head 2024-04-27T01:16:35.498 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d1c21232:::155:head 2024-04-27T01:16:35.498 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:a7a3c575:::156:head 2024-04-27T01:16:35.498 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:87da0633:::157:head 2024-04-27T01:16:35.498 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d5ac3822:::158:head 2024-04-27T01:16:35.498 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:3f20522d:::159:head 2024-04-27T01:16:35.498 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:6ca26563:::160:head 2024-04-27T01:16:35.498 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:532ce135:::161:head 2024-04-27T01:16:35.498 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:c78863e6:::162:head 2024-04-27T01:16:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-74325-57"}]': finished 2024-04-27T01:16:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-27T01:16:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: osdmap e308: 8 total, 8 up, 8 in 2024-04-27T01:16:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-74325-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:35.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-74325-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:35.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-74551-43","var": "pg_num","format": "json"}]: dispatch 2024-04-27T01:16:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-74325-57"}]': finished 2024-04-27T01:16:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-27T01:16:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: osdmap e308: 8 total, 8 up, 8 in 2024-04-27T01:16:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-74325-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-74325-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-74551-43","var": "pg_num","format": "json"}]: dispatch 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3338028583' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi162-74325-57"}]': finished 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: osdmap e308: 8 total, 8 up, 8 in 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-74325-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-74325-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:35.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi162-74551-43","var": "pg_num","format": "json"}]: dispatch 2024-04-27T01:16:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:36 smithi181 ceph-mon[30237]: pgmap v419: 228 pgs: 228 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:36 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43"}]: dispatch 2024-04-27T01:16:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:36 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43"}]: dispatch 2024-04-27T01:16:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:36 smithi181 ceph-mon[30237]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-74325-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:36 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43"}]': finished 2024-04-27T01:16:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:36 smithi181 ceph-mon[30237]: osdmap e309: 8 total, 8 up, 8 in 2024-04-27T01:16:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-74325-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:36.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:36 smithi181 ceph-mon[30237]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-74325-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[37680]: pgmap v419: 228 pgs: 228 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43"}]: dispatch 2024-04-27T01:16:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43"}]: dispatch 2024-04-27T01:16:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[37680]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-74325-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43"}]': finished 2024-04-27T01:16:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[37680]: osdmap e309: 8 total, 8 up, 8 in 2024-04-27T01:16:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-74325-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[37680]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-74325-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[26714]: pgmap v419: 228 pgs: 228 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43"}]: dispatch 2024-04-27T01:16:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43"}]: dispatch 2024-04-27T01:16:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[26714]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi162-74325-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-43"}]': finished 2024-04-27T01:16:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[26714]: osdmap e309: 8 total, 8 up, 8 in 2024-04-27T01:16:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-74325-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:36 smithi162 ceph-mon[26714]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-74325-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:38.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:38 smithi181 ceph-mon[30237]: pgmap v422: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 27 KiB/s wr, 56 op/s 2024-04-27T01:16:38.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:38 smithi181 ceph-mon[30237]: osdmap e310: 8 total, 8 up, 8 in 2024-04-27T01:16:38.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:38 smithi162 ceph-mon[37680]: pgmap v422: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 27 KiB/s wr, 56 op/s 2024-04-27T01:16:38.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:38 smithi162 ceph-mon[37680]: osdmap e310: 8 total, 8 up, 8 in 2024-04-27T01:16:38.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:38 smithi162 ceph-mon[26714]: pgmap v422: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 27 KiB/s wr, 56 op/s 2024-04-27T01:16:38.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:38 smithi162 ceph-mon[26714]: osdmap e310: 8 total, 8 up, 8 in 2024-04-27T01:16:39.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-mon[37680]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-74325-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-74325-58"}]': finished 2024-04-27T01:16:39.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-mon[37680]: osdmap e311: 8 total, 8 up, 8 in 2024-04-27T01:16:39.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:39.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:39.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:39.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-mon[26714]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-74325-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-74325-58"}]': finished 2024-04-27T01:16:39.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-mon[26714]: osdmap e311: 8 total, 8 up, 8 in 2024-04-27T01:16:39.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:39.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:39.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:39.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:39 smithi181 ceph-mon[30237]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi162-74325-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi162-74325-58"}]': finished 2024-04-27T01:16:39.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:39 smithi181 ceph-mon[30237]: osdmap e311: 8 total, 8 up, 8 in 2024-04-27T01:16:39.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:39.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:39.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:39 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:40.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:16:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:16:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:16:40.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:40 smithi162 ceph-mon[26714]: pgmap v425: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:40.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:40.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:40 smithi162 ceph-mon[26714]: osdmap e312: 8 total, 8 up, 8 in 2024-04-27T01:16:40.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:40.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:40.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:40 smithi162 ceph-mon[37680]: pgmap v425: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:40.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:40.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:40 smithi162 ceph-mon[37680]: osdmap e312: 8 total, 8 up, 8 in 2024-04-27T01:16:40.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:40.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:40.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:40 smithi181 ceph-mon[30237]: pgmap v425: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:40.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:40.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:40 smithi181 ceph-mon[30237]: osdmap e312: 8 total, 8 up, 8 in 2024-04-27T01:16:40.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:40.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:16:41.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:41.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-27T01:16:41.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[26714]: osdmap e313: 8 total, 8 up, 8 in 2024-04-27T01:16:41.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-27T01:16:41.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:41.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[26714]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:41.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:41.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-27T01:16:41.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[37680]: osdmap e313: 8 total, 8 up, 8 in 2024-04-27T01:16:41.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-27T01:16:41.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:41.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:41 smithi162 ceph-mon[37680]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:41.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:41 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:16:41.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-27T01:16:41.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:41 smithi181 ceph-mon[30237]: osdmap e313: 8 total, 8 up, 8 in 2024-04-27T01:16:41.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:41 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-27T01:16:41.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:41.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:41 smithi181 ceph-mon[30237]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:42.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[26714]: pgmap v428: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:42.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_count","val": "3"}]': finished 2024-04-27T01:16:42.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[26714]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]': finished 2024-04-27T01:16:42.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-27T01:16:42.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[26714]: osdmap e314: 8 total, 8 up, 8 in 2024-04-27T01:16:42.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:42.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-27T01:16:42.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[26714]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[37680]: pgmap v428: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_count","val": "3"}]': finished 2024-04-27T01:16:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[37680]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]': finished 2024-04-27T01:16:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-27T01:16:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[37680]: osdmap e314: 8 total, 8 up, 8 in 2024-04-27T01:16:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-27T01:16:42.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:42 smithi162 ceph-mon[37680]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:42.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:16:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:16:42.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:42 smithi181 ceph-mon[30237]: pgmap v428: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:42.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:42 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_count","val": "3"}]': finished 2024-04-27T01:16:42.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:42 smithi181 ceph-mon[30237]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi162-74325-58"}]': finished 2024-04-27T01:16:42.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-27T01:16:42.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:42 smithi181 ceph-mon[30237]: osdmap e314: 8 total, 8 up, 8 in 2024-04-27T01:16:42.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/842983690' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:42.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:42 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-27T01:16:42.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:42 smithi181 ceph-mon[30237]: from='client.49237 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]: dispatch 2024-04-27T01:16:43.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_period","val": "3"}]': finished 2024-04-27T01:16:43.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[37680]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]': finished 2024-04-27T01:16:43.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[37680]: osdmap e315: 8 total, 8 up, 8 in 2024-04-27T01:16:43.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:16:43.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:16:43.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:43.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:43.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:43.588 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:43.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_period","val": "3"}]': finished 2024-04-27T01:16:43.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[26714]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]': finished 2024-04-27T01:16:43.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[26714]: osdmap e315: 8 total, 8 up, 8 in 2024-04-27T01:16:43.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:16:43.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:16:43.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:43.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:43.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:43.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:43 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_period","val": "3"}]': finished 2024-04-27T01:16:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:43 smithi181 ceph-mon[30237]: from='client.49237 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi162-74325-58"}]': finished 2024-04-27T01:16:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:43 smithi181 ceph-mon[30237]: osdmap e315: 8 total, 8 up, 8 in 2024-04-27T01:16:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:16:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:43 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:16:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:43.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[37680]: pgmap v431: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:16:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-27T01:16:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[37680]: osdmap e316: 8 total, 8 up, 8 in 2024-04-27T01:16:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-74325-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:44.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-27T01:16:44.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[26714]: pgmap v431: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:44.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:16:44.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:44.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-27T01:16:44.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[26714]: osdmap e316: 8 total, 8 up, 8 in 2024-04-27T01:16:44.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-74325-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:44.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:44 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-27T01:16:45.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:44 smithi181 ceph-mon[30237]: pgmap v431: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:45.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:44 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:16:45.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:45.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-27T01:16:45.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:44 smithi181 ceph-mon[30237]: osdmap e316: 8 total, 8 up, 8 in 2024-04-27T01:16:45.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-74325-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:45.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:44 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-27T01:16:45.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:45 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-27T01:16:45.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:45 smithi162 ceph-mon[37680]: osdmap e317: 8 total, 8 up, 8 in 2024-04-27T01:16:45.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-74325-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]': finished 2024-04-27T01:16:45.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:45 smithi162 ceph-mon[37680]: osdmap e318: 8 total, 8 up, 8 in 2024-04-27T01:16:45.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:45 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-27T01:16:45.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:45 smithi162 ceph-mon[26714]: osdmap e317: 8 total, 8 up, 8 in 2024-04-27T01:16:45.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-74325-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]': finished 2024-04-27T01:16:45.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:45 smithi162 ceph-mon[26714]: osdmap e318: 8 total, 8 up, 8 in 2024-04-27T01:16:46.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:45 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-27T01:16:46.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:45 smithi181 ceph-mon[30237]: osdmap e317: 8 total, 8 up, 8 in 2024-04-27T01:16:46.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi162-74325-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]': finished 2024-04-27T01:16:46.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:45 smithi181 ceph-mon[30237]: osdmap e318: 8 total, 8 up, 8 in 2024-04-27T01:16:46.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:46 smithi162 ceph-mon[37680]: pgmap v434: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:46.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:46 smithi162 ceph-mon[37680]: osdmap e319: 8 total, 8 up, 8 in 2024-04-27T01:16:46.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:46 smithi162 ceph-mon[26714]: pgmap v434: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:46.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:46 smithi162 ceph-mon[26714]: osdmap e319: 8 total, 8 up, 8 in 2024-04-27T01:16:47.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:46 smithi181 ceph-mon[30237]: pgmap v434: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:16:47.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:46 smithi181 ceph-mon[30237]: osdmap e319: 8 total, 8 up, 8 in 2024-04-27T01:16:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:48 smithi162 ceph-mon[37680]: pgmap v437: 236 pgs: 2 unknown, 234 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:16:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:48 smithi162 ceph-mon[37680]: osdmap e320: 8 total, 8 up, 8 in 2024-04-27T01:16:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:48.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:48 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:48.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:48 smithi162 ceph-mon[26714]: pgmap v437: 236 pgs: 2 unknown, 234 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:16:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:48 smithi162 ceph-mon[26714]: osdmap e320: 8 total, 8 up, 8 in 2024-04-27T01:16:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:48.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:48 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:48.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:48 smithi181 ceph-mon[30237]: pgmap v437: 236 pgs: 2 unknown, 234 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:16:48.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:48 smithi181 ceph-mon[30237]: osdmap e320: 8 total, 8 up, 8 in 2024-04-27T01:16:48.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:48.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:48 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:49.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]': finished 2024-04-27T01:16:49.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-mon[37680]: osdmap e321: 8 total, 8 up, 8 in 2024-04-27T01:16:49.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:49.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-74325-59"}]': finished 2024-04-27T01:16:49.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-mon[37680]: osdmap e322: 8 total, 8 up, 8 in 2024-04-27T01:16:49.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]': finished 2024-04-27T01:16:49.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-mon[26714]: osdmap e321: 8 total, 8 up, 8 in 2024-04-27T01:16:49.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:49.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-74325-59"}]': finished 2024-04-27T01:16:49.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-mon[26714]: osdmap e322: 8 total, 8 up, 8 in 2024-04-27T01:16:49.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi162-74325-59"}]': finished 2024-04-27T01:16:49.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:49 smithi181 ceph-mon[30237]: osdmap e321: 8 total, 8 up, 8 in 2024-04-27T01:16:49.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-74325-59"}]: dispatch 2024-04-27T01:16:49.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1355689184' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi162-74325-59"}]': finished 2024-04-27T01:16:49.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:49 smithi181 ceph-mon[30237]: osdmap e322: 8 total, 8 up, 8 in 2024-04-27T01:16:50.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:16:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:16:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:16:50.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:50 smithi162 ceph-mon[37680]: pgmap v440: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-04-27T01:16:50.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:50.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:50 smithi162 ceph-mon[26714]: pgmap v440: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-04-27T01:16:50.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:50.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:50 smithi181 ceph-mon[30237]: pgmap v440: 228 pgs: 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-04-27T01:16:50.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:16:51.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:51 smithi181 ceph-mon[30237]: osdmap e323: 8 total, 8 up, 8 in 2024-04-27T01:16:51.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/972468428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:51.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:51 smithi181 ceph-mon[30237]: from='client.56999 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:51.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:51 smithi162 ceph-mon[37680]: osdmap e323: 8 total, 8 up, 8 in 2024-04-27T01:16:51.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/972468428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:51.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:51 smithi162 ceph-mon[37680]: from='client.56999 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:51.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:51 smithi162 ceph-mon[26714]: osdmap e323: 8 total, 8 up, 8 in 2024-04-27T01:16:51.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/972468428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:51.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:51 smithi162 ceph-mon[26714]: from='client.56999 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:16:52.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:7462ddf6.RoundTripAppendPP (3228 ms) 2024-04-27T01:16:52.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-04-27T01:16:52.356 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2779 ms) 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3021 ms) 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2956 ms) 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: setting pool EIO 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: max_success 76, min_failed 77 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4021 ms) 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3028 ms) 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (129494 ms total) 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-04-27T01:16:52.357 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3044 ms) 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9312 ms) 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14630 ms) 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3011 ms) 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29997 ms total) 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3249 ms) 2024-04-27T01:16:52.358 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3249 ms total) 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13730 ms) 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7032 ms) 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6977 ms) 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7044 ms) 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3023 ms) 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-04-27T01:16:52.359 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7024 ms) 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6908 ms) 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7048 ms) 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7041 ms) 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7002 ms) 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7027 ms) 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7041 ms) 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7021 ms) 2024-04-27T01:16:52.360 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-04-27T01:16:52.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:16:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:16:52.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:52 smithi181 ceph-mon[30237]: pgmap v443: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:52.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:52 smithi181 ceph-mon[30237]: from='client.56999 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:52.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:52 smithi181 ceph-mon[30237]: osdmap e324: 8 total, 8 up, 8 in 2024-04-27T01:16:52.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:52 smithi162 ceph-mon[37680]: pgmap v443: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:52.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:52 smithi162 ceph-mon[37680]: from='client.56999 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:52.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:52 smithi162 ceph-mon[37680]: osdmap e324: 8 total, 8 up, 8 in 2024-04-27T01:16:52.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:52 smithi162 ceph-mon[26714]: pgmap v443: 260 pgs: 32 unknown, 228 active+clean; 8.4 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:16:52.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:52 smithi162 ceph-mon[26714]: from='client.56999 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi162-74325-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:16:52.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:52 smithi162 ceph-mon[26714]: osdmap e324: 8 total, 8 up, 8 in 2024-04-27T01:16:53.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:53 smithi181 ceph-mon[30237]: osdmap e325: 8 total, 8 up, 8 in 2024-04-27T01:16:53.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:53 smithi181 ceph-mon[30237]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:53 smithi181 ceph-mon[30237]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-74325-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:53.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:53 smithi181 ceph-mon[30237]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-74325-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:53.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:53.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[37680]: osdmap e325: 8 total, 8 up, 8 in 2024-04-27T01:16:53.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[37680]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[37680]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-74325-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:53.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[37680]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-74325-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:53.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:53.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[26714]: osdmap e325: 8 total, 8 up, 8 in 2024-04-27T01:16:53.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[26714]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[26714]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:53.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-74325-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:53.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[26714]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-74325-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:16:53.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:16:55.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:54 smithi181 ceph-mon[30237]: pgmap v446: 228 pgs: 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-27T01:16:55.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:54 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:55.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:54 smithi181 ceph-mon[30237]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-74325-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:55.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:54 smithi181 ceph-mon[30237]: osdmap e326: 8 total, 8 up, 8 in 2024-04-27T01:16:55.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-74325-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:55.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:54 smithi181 ceph-mon[30237]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-74325-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:55.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[37680]: pgmap v446: 228 pgs: 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-27T01:16:55.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:55.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[37680]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-74325-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:55.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[37680]: osdmap e326: 8 total, 8 up, 8 in 2024-04-27T01:16:55.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-74325-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:55.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[37680]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-74325-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:55.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[26714]: pgmap v446: 228 pgs: 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-27T01:16:55.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:16:55.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[26714]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi162-74325-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:16:55.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[26714]: osdmap e326: 8 total, 8 up, 8 in 2024-04-27T01:16:55.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-74325-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:55.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:54 smithi162 ceph-mon[26714]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-74325-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:56.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:55 smithi181 ceph-mon[30237]: osdmap e327: 8 total, 8 up, 8 in 2024-04-27T01:16:56.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:55 smithi181 ceph-mon[30237]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-74325-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-74325-61"}]': finished 2024-04-27T01:16:56.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:55 smithi181 ceph-mon[30237]: osdmap e328: 8 total, 8 up, 8 in 2024-04-27T01:16:56.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:55 smithi162 ceph-mon[37680]: osdmap e327: 8 total, 8 up, 8 in 2024-04-27T01:16:56.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:55 smithi162 ceph-mon[37680]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-74325-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-74325-61"}]': finished 2024-04-27T01:16:56.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:55 smithi162 ceph-mon[37680]: osdmap e328: 8 total, 8 up, 8 in 2024-04-27T01:16:56.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:55 smithi162 ceph-mon[26714]: osdmap e327: 8 total, 8 up, 8 in 2024-04-27T01:16:56.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:55 smithi162 ceph-mon[26714]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi162-74325-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi162-74325-61"}]': finished 2024-04-27T01:16:56.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:55 smithi162 ceph-mon[26714]: osdmap e328: 8 total, 8 up, 8 in 2024-04-27T01:16:57.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:56 smithi181 ceph-mon[30237]: pgmap v449: 228 pgs: 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-27T01:16:57.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:57.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:56 smithi181 ceph-mon[30237]: osdmap e329: 8 total, 8 up, 8 in 2024-04-27T01:16:57.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:57.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45"}]: dispatch 2024-04-27T01:16:57.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45"}]: dispatch 2024-04-27T01:16:57.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[37680]: pgmap v449: 228 pgs: 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-27T01:16:57.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:57.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[37680]: osdmap e329: 8 total, 8 up, 8 in 2024-04-27T01:16:57.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:57.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45"}]: dispatch 2024-04-27T01:16:57.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45"}]: dispatch 2024-04-27T01:16:57.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[26714]: pgmap v449: 228 pgs: 228 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-27T01:16:57.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:57.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[26714]: osdmap e329: 8 total, 8 up, 8 in 2024-04-27T01:16:57.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:16:57.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45"}]: dispatch 2024-04-27T01:16:57.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45"}]: dispatch 2024-04-27T01:16:59.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:58 smithi181 ceph-mon[30237]: pgmap v452: 236 pgs: 4 unknown, 232 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-27T01:16:59.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45"}]': finished 2024-04-27T01:16:59.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:58 smithi181 ceph-mon[30237]: osdmap e330: 8 total, 8 up, 8 in 2024-04-27T01:16:59.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:59.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:58 smithi181 ceph-mon[30237]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:59.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:58 smithi162 ceph-mon[37680]: pgmap v452: 236 pgs: 4 unknown, 232 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-27T01:16:59.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45"}]': finished 2024-04-27T01:16:59.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:58 smithi162 ceph-mon[37680]: osdmap e330: 8 total, 8 up, 8 in 2024-04-27T01:16:59.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:59.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:58 smithi162 ceph-mon[37680]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:59.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:58 smithi162 ceph-mon[26714]: pgmap v452: 236 pgs: 4 unknown, 232 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-27T01:16:59.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-45"}]': finished 2024-04-27T01:16:59.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:58 smithi162 ceph-mon[26714]: osdmap e330: 8 total, 8 up, 8 in 2024-04-27T01:16:59.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:16:59.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:58 smithi162 ceph-mon[26714]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:17:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]': finished 2024-04-27T01:17:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: osdmap e331: 8 total, 8 up, 8 in 2024-04-27T01:17:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:17:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:17:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]': finished 2024-04-27T01:17:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: osdmap e332: 8 total, 8 up, 8 in 2024-04-27T01:17:00.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:00.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:16:59 smithi181 ceph-mon[30237]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:00.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:16:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:17:00.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]': finished 2024-04-27T01:17:00.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: osdmap e331: 8 total, 8 up, 8 in 2024-04-27T01:17:00.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:17:00.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:17:00.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:00.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]': finished 2024-04-27T01:17:00.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: osdmap e332: 8 total, 8 up, 8 in 2024-04-27T01:17:00.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[26714]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi162-74325-61"}]': finished 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: osdmap e331: 8 total, 8 up, 8 in 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/882362371' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.57005 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]: dispatch 2024-04-27T01:17:00.088 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.57005 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi162-74325-61"}]': finished 2024-04-27T01:17:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: osdmap e332: 8 total, 8 up, 8 in 2024-04-27T01:17:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:00.089 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:16:59 smithi162 ceph-mon[37680]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:01.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:00 smithi181 ceph-mon[30237]: pgmap v455: 196 pgs: 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:01.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:00 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:01.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:00 smithi181 ceph-mon[30237]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:01.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:00 smithi181 ceph-mon[30237]: osdmap e333: 8 total, 8 up, 8 in 2024-04-27T01:17:01.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-74325-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:01.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:00 smithi181 ceph-mon[30237]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-74325-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:01.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:01.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:00 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:01.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[37680]: pgmap v455: 196 pgs: 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:01.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:01.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[37680]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:01.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[37680]: osdmap e333: 8 total, 8 up, 8 in 2024-04-27T01:17:01.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-74325-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:01.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[37680]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-74325-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:01.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:01.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:01.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[26714]: pgmap v455: 196 pgs: 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:01.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:01.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[26714]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:01.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[26714]: osdmap e333: 8 total, 8 up, 8 in 2024-04-27T01:17:01.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-74325-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:01.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[26714]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-74325-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:01.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:01.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:00 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:02.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:17:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:17:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:02 smithi181 ceph-mon[30237]: pgmap v458: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:17:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:02 smithi181 ceph-mon[30237]: osdmap e334: 8 total, 8 up, 8 in 2024-04-27T01:17:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:02.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:02 smithi162 ceph-mon[37680]: pgmap v458: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:02.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:17:02.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:02.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:02 smithi162 ceph-mon[37680]: osdmap e334: 8 total, 8 up, 8 in 2024-04-27T01:17:02.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:02 smithi162 ceph-mon[26714]: pgmap v458: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:17:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:02 smithi162 ceph-mon[26714]: osdmap e334: 8 total, 8 up, 8 in 2024-04-27T01:17:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:03.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:03 smithi181 ceph-mon[30237]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-74325-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-74325-62"}]': finished 2024-04-27T01:17:03.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-47"}]': finished 2024-04-27T01:17:03.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:03 smithi181 ceph-mon[30237]: osdmap e335: 8 total, 8 up, 8 in 2024-04-27T01:17:03.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-47", "mode": "writeback"}]: dispatch 2024-04-27T01:17:03.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-47", "mode": "writeback"}]: dispatch 2024-04-27T01:17:03.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:03.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[37680]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-74325-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-74325-62"}]': finished 2024-04-27T01:17:03.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-47"}]': finished 2024-04-27T01:17:03.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[37680]: osdmap e335: 8 total, 8 up, 8 in 2024-04-27T01:17:03.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-47", "mode": "writeback"}]: dispatch 2024-04-27T01:17:03.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-47", "mode": "writeback"}]: dispatch 2024-04-27T01:17:03.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[26714]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi162-74325-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi162-74325-62"}]': finished 2024-04-27T01:17:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-47"}]': finished 2024-04-27T01:17:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[26714]: osdmap e335: 8 total, 8 up, 8 in 2024-04-27T01:17:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-47", "mode": "writeback"}]: dispatch 2024-04-27T01:17:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-47", "mode": "writeback"}]: dispatch 2024-04-27T01:17:03.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:04 smithi181 ceph-mon[30237]: pgmap v461: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-27T01:17:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:04 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:17:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-47", "mode": "writeback"}]': finished 2024-04-27T01:17:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:04 smithi181 ceph-mon[30237]: osdmap e336: 8 total, 8 up, 8 in 2024-04-27T01:17:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[37680]: pgmap v461: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-27T01:17:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:17:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-47", "mode": "writeback"}]': finished 2024-04-27T01:17:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[37680]: osdmap e336: 8 total, 8 up, 8 in 2024-04-27T01:17:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[26714]: pgmap v461: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-27T01:17:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:17:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-47", "mode": "writeback"}]': finished 2024-04-27T01:17:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[26714]: osdmap e336: 8 total, 8 up, 8 in 2024-04-27T01:17:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:05.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:17:05.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: osdmap e337: 8 total, 8 up, 8 in 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]': finished 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: osdmap e338: 8 total, 8 up, 8 in 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:05 smithi181 ceph-mon[30237]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:17:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: osdmap e337: 8 total, 8 up, 8 in 2024-04-27T01:17:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:05.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]': finished 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: osdmap e338: 8 total, 8 up, 8 in 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[37680]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: osdmap e337: 8 total, 8 up, 8 in 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:17:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi162-74325-62"}]': finished 2024-04-27T01:17:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: osdmap e338: 8 total, 8 up, 8 in 2024-04-27T01:17:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3663367999' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:05.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:05 smithi162 ceph-mon[26714]: from='client.57011 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]: dispatch 2024-04-27T01:17:06.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:06 smithi181 ceph-mon[30237]: pgmap v464: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-27T01:17:06.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:06 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:06.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:06 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:06.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:06 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:17:06.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:06 smithi181 ceph-mon[30237]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]': finished 2024-04-27T01:17:06.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:06 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:17:06.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:06.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:06 smithi181 ceph-mon[30237]: osdmap e339: 8 total, 8 up, 8 in 2024-04-27T01:17:06.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:06 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:06.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[26714]: pgmap v464: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-27T01:17:06.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:06.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:06.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:17:06.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[26714]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]': finished 2024-04-27T01:17:06.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:17:06.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:06.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[26714]: osdmap e339: 8 total, 8 up, 8 in 2024-04-27T01:17:06.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[37680]: pgmap v464: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-27T01:17:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:06.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:17:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[37680]: from='client.57011 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi162-74325-62"}]': finished 2024-04-27T01:17:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:17:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[37680]: osdmap e339: 8 total, 8 up, 8 in 2024-04-27T01:17:06.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:06 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:07.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-74325-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:07.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-74325-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:07.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:17:07.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-74325-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:07.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: osdmap e340: 8 total, 8 up, 8 in 2024-04-27T01:17:07.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-27T01:17:07.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:07 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-74325-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-74325-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-74325-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: osdmap e340: 8 total, 8 up, 8 in 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-27T01:17:07.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:07.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-74325-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:07.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-74325-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:07.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:17:07.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi162-74325-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:07.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: osdmap e340: 8 total, 8 up, 8 in 2024-04-27T01:17:07.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-27T01:17:07.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:07 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-27T01:17:09.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:08 smithi181 ceph-mon[30237]: pgmap v467: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:09.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-74325-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:09.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:08 smithi181 ceph-mon[30237]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-74325-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:09.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-27T01:17:09.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:08 smithi162 ceph-mon[37680]: pgmap v467: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:09.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-74325-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:09.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:08 smithi162 ceph-mon[37680]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-74325-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:09.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-27T01:17:09.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:08 smithi162 ceph-mon[26714]: pgmap v467: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:09.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-74325-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:09.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:08 smithi162 ceph-mon[26714]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-74325-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:09.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-27T01:17:10.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-27T01:17:10.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:09 smithi181 ceph-mon[30237]: osdmap e341: 8 total, 8 up, 8 in 2024-04-27T01:17:10.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-27T01:17:10.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:09 smithi181 ceph-mon[30237]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-74325-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-74325-63"}]': finished 2024-04-27T01:17:10.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-27T01:17:10.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:09 smithi181 ceph-mon[30237]: osdmap e342: 8 total, 8 up, 8 in 2024-04-27T01:17:10.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:10.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:10.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:17:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:17:10.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-27T01:17:10.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[37680]: osdmap e341: 8 total, 8 up, 8 in 2024-04-27T01:17:10.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-27T01:17:10.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[37680]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-74325-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-74325-63"}]': finished 2024-04-27T01:17:10.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-27T01:17:10.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[37680]: osdmap e342: 8 total, 8 up, 8 in 2024-04-27T01:17:10.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:10.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:10.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-27T01:17:10.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[26714]: osdmap e341: 8 total, 8 up, 8 in 2024-04-27T01:17:10.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-27T01:17:10.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[26714]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi162-74325-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi162-74325-63"}]': finished 2024-04-27T01:17:10.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-27T01:17:10.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[26714]: osdmap e342: 8 total, 8 up, 8 in 2024-04-27T01:17:10.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:10.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:11.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:10 smithi181 ceph-mon[30237]: pgmap v470: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:11.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:17:11.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:11.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:10 smithi181 ceph-mon[30237]: osdmap e343: 8 total, 8 up, 8 in 2024-04-27T01:17:11.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:11.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:10 smithi162 ceph-mon[37680]: pgmap v470: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:11.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:17:11.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:11.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:10 smithi162 ceph-mon[37680]: osdmap e343: 8 total, 8 up, 8 in 2024-04-27T01:17:11.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:11.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:10 smithi162 ceph-mon[26714]: pgmap v470: 228 pgs: 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:11.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:17:11.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:11.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:10 smithi162 ceph-mon[26714]: osdmap e343: 8 total, 8 up, 8 in 2024-04-27T01:17:11.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:12.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:17:12.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-mon[30237]: pgmap v473: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:12.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]': finished 2024-04-27T01:17:12.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-mon[30237]: osdmap e344: 8 total, 8 up, 8 in 2024-04-27T01:17:12.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:12.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-mon[30237]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:12.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:12.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:12.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:12.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:12.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:12 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:12.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[26714]: pgmap v473: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:12.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]': finished 2024-04-27T01:17:12.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[26714]: osdmap e344: 8 total, 8 up, 8 in 2024-04-27T01:17:12.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:12.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[26714]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:12.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:12.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:12.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[37680]: pgmap v473: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]': finished 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[37680]: osdmap e344: 8 total, 8 up, 8 in 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[37680]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:12.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-47"}]: dispatch 2024-04-27T01:17:12.839 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:12 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:13.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:13 smithi181 ceph-mon[30237]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]': finished 2024-04-27T01:17:13.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:13 smithi181 ceph-mon[30237]: osdmap e345: 8 total, 8 up, 8 in 2024-04-27T01:17:13.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:13.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:13 smithi181 ceph-mon[30237]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:13.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:13.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:13 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:17:13.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:13 smithi181 ceph-mon[30237]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]': finished 2024-04-27T01:17:13.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[37680]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]': finished 2024-04-27T01:17:13.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[37680]: osdmap e345: 8 total, 8 up, 8 in 2024-04-27T01:17:13.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:13.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[37680]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:13.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:13.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:17:13.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[37680]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]': finished 2024-04-27T01:17:13.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[26714]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi162-74325-63"}]': finished 2024-04-27T01:17:13.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[26714]: osdmap e345: 8 total, 8 up, 8 in 2024-04-27T01:17:13.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/521031910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:13.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[26714]: from='client.57017 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]: dispatch 2024-04-27T01:17:13.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:13.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:17:13.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:13 smithi162 ceph-mon[26714]: from='client.57017 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi162-74325-63"}]': finished 2024-04-27T01:17:14.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: pgmap v476: 196 pgs: 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:17:14.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: osdmap e346: 8 total, 8 up, 8 in 2024-04-27T01:17:14.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:14.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:14.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-74325-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:14.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-74325-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:14.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:17:14.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:17:14.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:14.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:14.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-74325-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:14.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: osdmap e347: 8 total, 8 up, 8 in 2024-04-27T01:17:14.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-74325-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:14 smithi181 ceph-mon[30237]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-74325-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: pgmap v476: 196 pgs: 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:17:14.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: osdmap e346: 8 total, 8 up, 8 in 2024-04-27T01:17:14.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:14.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:14.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-74325-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:14.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-74325-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:14.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:17:14.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:17:14.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:14.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:14.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-74325-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:14.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: osdmap e347: 8 total, 8 up, 8 in 2024-04-27T01:17:14.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-74325-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[37680]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-74325-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: pgmap v476: 196 pgs: 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: osdmap e346: 8 total, 8 up, 8 in 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-74325-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-74325-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:14.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:14.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi162-74325-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:14.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: osdmap e347: 8 total, 8 up, 8 in 2024-04-27T01:17:14.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-74325-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:14.840 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:14 smithi162 ceph-mon[26714]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-74325-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:16.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:16.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:15 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:16.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:15 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:17:16.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:16.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:15 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:16.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:15 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:17:16.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:16.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:15 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:16.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:15 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:17:17.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:16 smithi181 ceph-mon[30237]: pgmap v479: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:17:17.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:17.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:16 smithi181 ceph-mon[30237]: osdmap e348: 8 total, 8 up, 8 in 2024-04-27T01:17:17.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:16 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:17.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:16 smithi181 ceph-mon[30237]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-74325-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-74325-64"}]': finished 2024-04-27T01:17:17.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:16 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-49"}]': finished 2024-04-27T01:17:17.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:16 smithi181 ceph-mon[30237]: osdmap e349: 8 total, 8 up, 8 in 2024-04-27T01:17:17.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-49", "mode": "readproxy"}]: dispatch 2024-04-27T01:17:17.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:16 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-49", "mode": "readproxy"}]: dispatch 2024-04-27T01:17:17.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[37680]: pgmap v479: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:17:17.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:17.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[37680]: osdmap e348: 8 total, 8 up, 8 in 2024-04-27T01:17:17.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:17.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[37680]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-74325-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-74325-64"}]': finished 2024-04-27T01:17:17.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-49"}]': finished 2024-04-27T01:17:17.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[37680]: osdmap e349: 8 total, 8 up, 8 in 2024-04-27T01:17:17.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-49", "mode": "readproxy"}]: dispatch 2024-04-27T01:17:17.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-49", "mode": "readproxy"}]: dispatch 2024-04-27T01:17:17.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[26714]: pgmap v479: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:17:17.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:17.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[26714]: osdmap e348: 8 total, 8 up, 8 in 2024-04-27T01:17:17.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:17.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[26714]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi162-74325-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi162-74325-64"}]': finished 2024-04-27T01:17:17.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-49"}]': finished 2024-04-27T01:17:17.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[26714]: osdmap e349: 8 total, 8 up, 8 in 2024-04-27T01:17:17.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-49", "mode": "readproxy"}]: dispatch 2024-04-27T01:17:17.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:16 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-49", "mode": "readproxy"}]: dispatch 2024-04-27T01:17:18.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:17 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:17:18.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:17 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-49", "mode": "readproxy"}]': finished 2024-04-27T01:17:18.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:17 smithi181 ceph-mon[30237]: osdmap e350: 8 total, 8 up, 8 in 2024-04-27T01:17:18.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:17 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:17:18.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:17 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-49", "mode": "readproxy"}]': finished 2024-04-27T01:17:18.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:17 smithi162 ceph-mon[37680]: osdmap e350: 8 total, 8 up, 8 in 2024-04-27T01:17:18.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:17 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:17:18.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:17 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-49", "mode": "readproxy"}]': finished 2024-04-27T01:17:18.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:17 smithi162 ceph-mon[26714]: osdmap e350: 8 total, 8 up, 8 in 2024-04-27T01:17:19.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:18 smithi181 ceph-mon[30237]: pgmap v482: 236 pgs: 3 creating+peering, 14 unknown, 219 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:17:19.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:18 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:19.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:18 smithi181 ceph-mon[30237]: osdmap e351: 8 total, 8 up, 8 in 2024-04-27T01:17:19.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:19.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:18 smithi181 ceph-mon[30237]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:19.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:18 smithi162 ceph-mon[37680]: pgmap v482: 236 pgs: 3 creating+peering, 14 unknown, 219 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:17:19.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:18 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:19.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:18 smithi162 ceph-mon[37680]: osdmap e351: 8 total, 8 up, 8 in 2024-04-27T01:17:19.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:19.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:18 smithi162 ceph-mon[37680]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:19.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:18 smithi162 ceph-mon[26714]: pgmap v482: 236 pgs: 3 creating+peering, 14 unknown, 219 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:17:19.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:18 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:19.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:18 smithi162 ceph-mon[26714]: osdmap e351: 8 total, 8 up, 8 in 2024-04-27T01:17:19.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:19.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:18 smithi162 ceph-mon[26714]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:20.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:17:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:17:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:17:20.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:20 smithi181 ceph-mon[30237]: pgmap v485: 228 pgs: 228 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:17:20.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:20 smithi181 ceph-mon[30237]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]': finished 2024-04-27T01:17:20.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:20 smithi181 ceph-mon[30237]: osdmap e352: 8 total, 8 up, 8 in 2024-04-27T01:17:20.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:20.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:20 smithi181 ceph-mon[30237]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:20.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:20.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[37680]: pgmap v485: 228 pgs: 228 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:17:20.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[37680]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]': finished 2024-04-27T01:17:20.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[37680]: osdmap e352: 8 total, 8 up, 8 in 2024-04-27T01:17:20.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:20.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[37680]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:20.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:20.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[26714]: pgmap v485: 228 pgs: 228 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:17:20.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[26714]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi162-74325-64"}]': finished 2024-04-27T01:17:20.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[26714]: osdmap e352: 8 total, 8 up, 8 in 2024-04-27T01:17:20.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4081910662' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:20.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[26714]: from='client.57023 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]: dispatch 2024-04-27T01:17:20.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:21.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:21 smithi181 ceph-mon[30237]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]': finished 2024-04-27T01:17:21.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:21 smithi181 ceph-mon[30237]: osdmap e353: 8 total, 8 up, 8 in 2024-04-27T01:17:21.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:21.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:21.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-74325-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:21.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:21 smithi162 ceph-mon[26714]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]': finished 2024-04-27T01:17:21.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:21 smithi162 ceph-mon[26714]: osdmap e353: 8 total, 8 up, 8 in 2024-04-27T01:17:21.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:21.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:21.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-74325-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:21.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:21 smithi162 ceph-mon[37680]: from='client.57023 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi162-74325-64"}]': finished 2024-04-27T01:17:21.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:21 smithi162 ceph-mon[37680]: osdmap e353: 8 total, 8 up, 8 in 2024-04-27T01:17:21.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:21.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:21.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-74325-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:22.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:17:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:17:23.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:22 smithi181 ceph-mon[30237]: pgmap v488: 228 pgs: 228 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:23.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-74325-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:23.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:22 smithi181 ceph-mon[30237]: osdmap e354: 8 total, 8 up, 8 in 2024-04-27T01:17:23.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-74325-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:23.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:22 smithi162 ceph-mon[37680]: pgmap v488: 228 pgs: 228 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:23.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-74325-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:23.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:22 smithi162 ceph-mon[37680]: osdmap e354: 8 total, 8 up, 8 in 2024-04-27T01:17:23.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-74325-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:23.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:22 smithi162 ceph-mon[26714]: pgmap v488: 228 pgs: 228 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:23.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi162-74325-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:23.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:22 smithi162 ceph-mon[26714]: osdmap e354: 8 total, 8 up, 8 in 2024-04-27T01:17:23.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-74325-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:24.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:24.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:23 smithi181 ceph-mon[30237]: osdmap e355: 8 total, 8 up, 8 in 2024-04-27T01:17:24.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-74325-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-74325-65"}]': finished 2024-04-27T01:17:24.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:23 smithi181 ceph-mon[30237]: osdmap e356: 8 total, 8 up, 8 in 2024-04-27T01:17:24.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:24.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:23 smithi162 ceph-mon[37680]: osdmap e355: 8 total, 8 up, 8 in 2024-04-27T01:17:24.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-74325-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-74325-65"}]': finished 2024-04-27T01:17:24.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:23 smithi162 ceph-mon[37680]: osdmap e356: 8 total, 8 up, 8 in 2024-04-27T01:17:24.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:24.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:23 smithi162 ceph-mon[26714]: osdmap e355: 8 total, 8 up, 8 in 2024-04-27T01:17:24.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi162-74325-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi162-74325-65"}]': finished 2024-04-27T01:17:24.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:23 smithi162 ceph-mon[26714]: osdmap e356: 8 total, 8 up, 8 in 2024-04-27T01:17:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:24 smithi181 ceph-mon[30237]: pgmap v491: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-27T01:17:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:24 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:25.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:24 smithi181 ceph-mon[30237]: osdmap e357: 8 total, 8 up, 8 in 2024-04-27T01:17:25.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:24 smithi162 ceph-mon[26714]: pgmap v491: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-27T01:17:25.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:24 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:25.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:24 smithi162 ceph-mon[26714]: osdmap e357: 8 total, 8 up, 8 in 2024-04-27T01:17:25.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:24 smithi162 ceph-mon[37680]: pgmap v491: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-27T01:17:25.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:24 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:25.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:24 smithi162 ceph-mon[37680]: osdmap e357: 8 total, 8 up, 8 in 2024-04-27T01:17:26.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:26 smithi181 ceph-mon[30237]: pgmap v494: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-27T01:17:26.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:26 smithi181 ceph-mon[30237]: osdmap e358: 8 total, 8 up, 8 in 2024-04-27T01:17:26.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:26.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:26 smithi162 ceph-mon[26714]: pgmap v494: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-27T01:17:26.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:26 smithi162 ceph-mon[26714]: osdmap e358: 8 total, 8 up, 8 in 2024-04-27T01:17:26.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:26.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:26 smithi162 ceph-mon[37680]: pgmap v494: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-27T01:17:26.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:26 smithi162 ceph-mon[37680]: osdmap e358: 8 total, 8 up, 8 in 2024-04-27T01:17:26.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:27.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-74325-65"}]': finished 2024-04-27T01:17:27.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:27 smithi181 ceph-mon[30237]: osdmap e359: 8 total, 8 up, 8 in 2024-04-27T01:17:27.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:27.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-74325-65"}]': finished 2024-04-27T01:17:27.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:27 smithi181 ceph-mon[30237]: osdmap e360: 8 total, 8 up, 8 in 2024-04-27T01:17:27.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-74325-65"}]': finished 2024-04-27T01:17:27.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:27 smithi162 ceph-mon[26714]: osdmap e359: 8 total, 8 up, 8 in 2024-04-27T01:17:27.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:27.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-74325-65"}]': finished 2024-04-27T01:17:27.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:27 smithi162 ceph-mon[26714]: osdmap e360: 8 total, 8 up, 8 in 2024-04-27T01:17:27.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi162-74325-65"}]': finished 2024-04-27T01:17:27.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:27 smithi162 ceph-mon[37680]: osdmap e359: 8 total, 8 up, 8 in 2024-04-27T01:17:27.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-74325-65"}]: dispatch 2024-04-27T01:17:27.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1999498047' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi162-74325-65"}]': finished 2024-04-27T01:17:27.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:27 smithi162 ceph-mon[37680]: osdmap e360: 8 total, 8 up, 8 in 2024-04-27T01:17:28.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:28 smithi181 ceph-mon[30237]: pgmap v497: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:28.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:28.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:28.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-74325-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:28.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:28.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:28.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[26714]: pgmap v497: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:28.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:28.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:28.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-74325-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:28.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:28.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:28.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[37680]: pgmap v497: 228 pgs: 228 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:28.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:28.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:28.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-74325-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:17:28.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:28.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:30.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-74325-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:30.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:29 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:17:30.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:30.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:29 smithi181 ceph-mon[30237]: osdmap e361: 8 total, 8 up, 8 in 2024-04-27T01:17:30.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-74325-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:30.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:29 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:30.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:29 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:17:30.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:29 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]': finished 2024-04-27T01:17:30.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:17:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:17:30.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-74325-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:30.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:17:30.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:30.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[37680]: osdmap e361: 8 total, 8 up, 8 in 2024-04-27T01:17:30.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-74325-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:30.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:30.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:17:30.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]': finished 2024-04-27T01:17:30.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi162-74325-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:17:30.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:17:30.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:30.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[26714]: osdmap e361: 8 total, 8 up, 8 in 2024-04-27T01:17:30.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-74325-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:30.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:30.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:17:30.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:29 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]': finished 2024-04-27T01:17:30.472 INFO:tasks.workunit.client.0.smithi162.stdout: :::163:head 2024-04-27T01:17:30.472 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:5d165639:::164:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:f43765fc:::165:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:b4c720e9:::166:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:e694b040:::167:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:afa38db2:::168:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:77ba9f53:::169:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:87495034:::170:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:7c96bf0e:::171:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:dbe346cc:::172:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:e943ec24:::173:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:f97a9c0c:::174:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:6f26e74d:::175:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:4f95e106:::176:head 2024-04-27T01:17:30.473 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:0e6f2f8f:::177:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:05db05f1:::178:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:38a78d66:::179:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:d095610b:::180:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:a1a9d709:::181:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:1e5d39db:::182:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:f7df4fb9:::183:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:03a7f161:::184:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:ba70721e:::185:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:28e5662d:::186:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:973d52de:::187:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:4303eb1c:::188:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:b990b48e:::189:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:29b8165b:::190:head 2024-04-27T01:17:30.474 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:3547f197:::191:head 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:7e260936:::192:head 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:1abec7b1:::193:head 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:10fdda93:::194:head 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:15817eea:::195:head 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:770bab57:::196:head 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:ed9e13e7:::197:head 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:71471a8f:::198:head 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: checking for 277:10fb1d02:::199:head 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8254 ms) 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,0 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: first is 1714180605 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,0 2024-04-27T01:17:30.475 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,1714180607,1714180608,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,1714180607,1714180608,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,1714180607,1714180608,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,1714180607,1714180608,1714180610,1714180611,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,1714180607,1714180608,1714180610,1714180611,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,1714180607,1714180608,1714180610,1714180611,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,1714180607,1714180608,1714180610,1714180611,1714180613,1714180614,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,1714180607,1714180608,1714180610,1714180611,1714180613,1714180614,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180605,1714180607,1714180608,1714180610,1714180611,1714180613,1714180614,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714180608,1714180610,1714180611,1714180613,1714180614,1714180616,1714180617,0 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: first now 1714180608, trimmed 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20873 ms) 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-04-27T01:17:30.476 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: foo0 2024-04-27T01:17:30.477 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-27T01:17:30.477 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14040 ms) 2024-04-27T01:17:30.477 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-04-27T01:17:31.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:30 smithi181 ceph-mon[30237]: pgmap v500: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:31.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:30 smithi181 ceph-mon[30237]: osdmap e362: 8 total, 8 up, 8 in 2024-04-27T01:17:31.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:31.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:30 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:31.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:31.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:30 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:31.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-74325-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-74325-66"}]': finished 2024-04-27T01:17:31.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:30 smithi181 ceph-mon[30237]: osdmap e363: 8 total, 8 up, 8 in 2024-04-27T01:17:31.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[37680]: pgmap v500: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:31.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[37680]: osdmap e362: 8 total, 8 up, 8 in 2024-04-27T01:17:31.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:31.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:31.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:31.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:31.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-74325-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-74325-66"}]': finished 2024-04-27T01:17:31.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[37680]: osdmap e363: 8 total, 8 up, 8 in 2024-04-27T01:17:31.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[26714]: pgmap v500: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:31.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[26714]: osdmap e362: 8 total, 8 up, 8 in 2024-04-27T01:17:31.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:31.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:31.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:31.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-49"}]: dispatch 2024-04-27T01:17:31.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi162-74325-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi162-74325-66"}]': finished 2024-04-27T01:17:31.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:30 smithi162 ceph-mon[26714]: osdmap e363: 8 total, 8 up, 8 in 2024-04-27T01:17:32.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:17:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:17:32.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:32 smithi181 ceph-mon[30237]: pgmap v503: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:32.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:32 smithi181 ceph-mon[30237]: osdmap e364: 8 total, 8 up, 8 in 2024-04-27T01:17:32.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:32.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:32 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:32.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:32 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:32.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:32 smithi162 ceph-mon[37680]: pgmap v503: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:32.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:32 smithi162 ceph-mon[37680]: osdmap e364: 8 total, 8 up, 8 in 2024-04-27T01:17:32.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:32.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:32 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:32.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:32 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:32.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:32 smithi162 ceph-mon[26714]: pgmap v503: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:17:32.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:32 smithi162 ceph-mon[26714]: osdmap e364: 8 total, 8 up, 8 in 2024-04-27T01:17:32.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:32.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:32 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:32.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:32 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:33.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:33 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:33 smithi181 ceph-mon[30237]: osdmap e365: 8 total, 8 up, 8 in 2024-04-27T01:17:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:33.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:33 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:33.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:33.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[37680]: osdmap e365: 8 total, 8 up, 8 in 2024-04-27T01:17:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:33.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:33.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:33.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:33.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[26714]: osdmap e365: 8 total, 8 up, 8 in 2024-04-27T01:17:33.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:33.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:33.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:33 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:17:34.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3037 ms) 2024-04-27T01:17:34.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-04-27T01:17:34.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7025 ms) 2024-04-27T01:17:34.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-04-27T01:17:34.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7022 ms) 2024-04-27T01:17:34.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-04-27T01:17:34.500 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7016 ms) 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7016 ms) 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7018 ms) 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7044 ms) 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139097 ms total) 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (301837 ms total) 2024-04-27T01:17:34.501 INFO:tasks.workunit.client.0.smithi162.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-04-27T01:17:34.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:34 smithi181 ceph-mon[30237]: pgmap v506: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-74325-66"}]': finished 2024-04-27T01:17:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:17:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:34 smithi181 ceph-mon[30237]: osdmap e366: 8 total, 8 up, 8 in 2024-04-27T01:17:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:34.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:34.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[37680]: pgmap v506: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-74325-66"}]': finished 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[37680]: osdmap e366: 8 total, 8 up, 8 in 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[26714]: pgmap v506: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi162-74325-66"}]': finished 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:34.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[26714]: osdmap e366: 8 total, 8 up, 8 in 2024-04-27T01:17:34.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-74325-66"}]: dispatch 2024-04-27T01:17:34.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:35.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-74325-66"}]': finished 2024-04-27T01:17:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:35 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-51"}]': finished 2024-04-27T01:17:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:35 smithi181 ceph-mon[30237]: osdmap e367: 8 total, 8 up, 8 in 2024-04-27T01:17:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-51", "mode": "writeback"}]: dispatch 2024-04-27T01:17:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:35 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-51", "mode": "writeback"}]: dispatch 2024-04-27T01:17:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:35 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:35.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:35 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:35.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-74325-66"}]': finished 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-51"}]': finished 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[37680]: osdmap e367: 8 total, 8 up, 8 in 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-51", "mode": "writeback"}]: dispatch 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-51", "mode": "writeback"}]: dispatch 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2942690712' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi162-74325-66"}]': finished 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-51"}]': finished 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[26714]: osdmap e367: 8 total, 8 up, 8 in 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-51", "mode": "writeback"}]: dispatch 2024-04-27T01:17:35.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-51", "mode": "writeback"}]: dispatch 2024-04-27T01:17:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:35.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:35 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:36.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[37680]: pgmap v509: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:17:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-51", "mode": "writeback"}]': finished 2024-04-27T01:17:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[37680]: osdmap e368: 8 total, 8 up, 8 in 2024-04-27T01:17:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:36.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:36.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[26714]: pgmap v509: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:36.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:17:36.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-51", "mode": "writeback"}]': finished 2024-04-27T01:17:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[26714]: osdmap e368: 8 total, 8 up, 8 in 2024-04-27T01:17:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:36.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:36 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:37.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:36 smithi181 ceph-mon[30237]: pgmap v509: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:17:37.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:36 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:17:37.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:36 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-51", "mode": "writeback"}]': finished 2024-04-27T01:17:37.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:36 smithi181 ceph-mon[30237]: osdmap e368: 8 total, 8 up, 8 in 2024-04-27T01:17:37.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:37.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:36 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:17:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:17:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[37680]: osdmap e369: 8 total, 8 up, 8 in 2024-04-27T01:17:37.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:37.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:17:37.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[37680]: osdmap e370: 8 total, 8 up, 8 in 2024-04-27T01:17:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:17:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[26714]: osdmap e369: 8 total, 8 up, 8 in 2024-04-27T01:17:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:17:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:37 smithi162 ceph-mon[26714]: osdmap e370: 8 total, 8 up, 8 in 2024-04-27T01:17:38.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:37 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:17:38.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:38.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:37 smithi181 ceph-mon[30237]: osdmap e369: 8 total, 8 up, 8 in 2024-04-27T01:17:38.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:37 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:17:38.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:37 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:17:38.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:37 smithi181 ceph-mon[30237]: osdmap e370: 8 total, 8 up, 8 in 2024-04-27T01:17:38.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[37680]: pgmap v512: 228 pgs: 10 creating+peering, 218 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-27T01:17:38.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[37680]: osdmap e371: 8 total, 8 up, 8 in 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[26714]: pgmap v512: 228 pgs: 10 creating+peering, 218 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:17:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:17:38.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[26714]: osdmap e371: 8 total, 8 up, 8 in 2024-04-27T01:17:38.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:38.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:38 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:39.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:38 smithi181 ceph-mon[30237]: pgmap v512: 228 pgs: 10 creating+peering, 218 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-04-27T01:17:39.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:39.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:38 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:17:39.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:38 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:17:39.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:38 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:17:39.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:38 smithi181 ceph-mon[30237]: osdmap e371: 8 total, 8 up, 8 in 2024-04-27T01:17:39.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:39.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:38 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:17:39.836 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:17:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:17:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:17:39.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:39 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:39.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:17:39.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:39 smithi162 ceph-mon[26714]: osdmap e372: 8 total, 8 up, 8 in 2024-04-27T01:17:39.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-27T01:17:39.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:39 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:39.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:17:39.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:39 smithi162 ceph-mon[37680]: osdmap e372: 8 total, 8 up, 8 in 2024-04-27T01:17:39.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-27T01:17:40.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:39 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:40.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:17:40.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:39 smithi181 ceph-mon[30237]: osdmap e372: 8 total, 8 up, 8 in 2024-04-27T01:17:40.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-27T01:17:40.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:40 smithi162 ceph-mon[37680]: pgmap v515: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:17:40.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-27T01:17:40.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "target_max_objects","val": "1"}]': finished 2024-04-27T01:17:40.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:40 smithi162 ceph-mon[37680]: osdmap e373: 8 total, 8 up, 8 in 2024-04-27T01:17:40.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:40 smithi162 ceph-mon[26714]: pgmap v515: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:17:40.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-27T01:17:40.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "target_max_objects","val": "1"}]': finished 2024-04-27T01:17:40.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:40 smithi162 ceph-mon[26714]: osdmap e373: 8 total, 8 up, 8 in 2024-04-27T01:17:41.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:40 smithi181 ceph-mon[30237]: pgmap v515: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:17:41.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-27T01:17:41.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-51","var": "target_max_objects","val": "1"}]': finished 2024-04-27T01:17:41.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:40 smithi181 ceph-mon[30237]: osdmap e373: 8 total, 8 up, 8 in 2024-04-27T01:17:41.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:41 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-27T01:17:41.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:41 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-27T01:17:42.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:41 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-27T01:17:42.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:17:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:17:42.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:42 smithi181 ceph-mon[30237]: pgmap v518: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-04-27T01:17:42.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:42 smithi162 ceph-mon[37680]: pgmap v518: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-04-27T01:17:42.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:42 smithi162 ceph-mon[26714]: pgmap v518: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 255 B/s wr, 0 op/s 2024-04-27T01:17:43.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:43.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:44.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:45.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:44 smithi181 ceph-mon[30237]: pgmap v519: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:45.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:44 smithi162 ceph-mon[37680]: pgmap v519: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:45.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:44 smithi162 ceph-mon[26714]: pgmap v519: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:47.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:46 smithi181 ceph-mon[30237]: pgmap v520: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 763 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:47.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:46 smithi162 ceph-mon[37680]: pgmap v520: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 763 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:47.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:46 smithi162 ceph-mon[26714]: pgmap v520: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 763 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:49.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:48 smithi181 ceph-mon[30237]: pgmap v521: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:49.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:48 smithi162 ceph-mon[37680]: pgmap v521: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:49.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:48 smithi162 ceph-mon[26714]: pgmap v521: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:50.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:17:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:17:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:17:51.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:50 smithi181 ceph-mon[30237]: pgmap v522: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:51.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:50 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:51.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:51.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:51.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:50 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:51.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:50 smithi162 ceph-mon[37680]: pgmap v522: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:51.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:50 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:51.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:51.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:51.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:50 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:51.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:50 smithi162 ceph-mon[26714]: pgmap v522: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:51.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:50 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:17:51.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:17:51.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:51.088 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:50 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:52.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:52 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:17:52.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:52.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:52 smithi181 ceph-mon[30237]: osdmap e374: 8 total, 8 up, 8 in 2024-04-27T01:17:52.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:52 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:52.451 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:52 smithi181 ceph-mon[30237]: pgmap v524: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:52.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:52 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:17:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:52 smithi162 ceph-mon[37680]: osdmap e374: 8 total, 8 up, 8 in 2024-04-27T01:17:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:52 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:52.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:52 smithi162 ceph-mon[37680]: pgmap v524: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:52.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:52 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:17:52.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:52.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:52 smithi162 ceph-mon[26714]: osdmap e374: 8 total, 8 up, 8 in 2024-04-27T01:17:52.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:52 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:52.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:52 smithi162 ceph-mon[26714]: pgmap v524: 228 pgs: 228 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:17:52.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:17:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:17:53.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:53 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]': finished 2024-04-27T01:17:53.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:53 smithi181 ceph-mon[30237]: osdmap e375: 8 total, 8 up, 8 in 2024-04-27T01:17:53.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:53.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:53 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:53.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:53.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:53 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:53.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:53.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]': finished 2024-04-27T01:17:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[37680]: osdmap e375: 8 total, 8 up, 8 in 2024-04-27T01:17:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:53.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:53.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]': finished 2024-04-27T01:17:53.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[26714]: osdmap e375: 8 total, 8 up, 8 in 2024-04-27T01:17:53.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:53.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-51"}]: dispatch 2024-04-27T01:17:53.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:17:54.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:54 smithi181 ceph-mon[30237]: osdmap e376: 8 total, 8 up, 8 in 2024-04-27T01:17:54.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:54 smithi181 ceph-mon[30237]: pgmap v527: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:17:54.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:54 smithi162 ceph-mon[26714]: osdmap e376: 8 total, 8 up, 8 in 2024-04-27T01:17:54.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:54 smithi162 ceph-mon[26714]: pgmap v527: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:17:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:54 smithi162 ceph-mon[37680]: osdmap e376: 8 total, 8 up, 8 in 2024-04-27T01:17:54.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:54 smithi162 ceph-mon[37680]: pgmap v527: 196 pgs: 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-27T01:17:55.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:55 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-27T01:17:55.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:55 smithi181 ceph-mon[30237]: osdmap e377: 8 total, 8 up, 8 in 2024-04-27T01:17:55.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:55.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:55 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:55.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:55 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-27T01:17:55.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:55 smithi162 ceph-mon[37680]: osdmap e377: 8 total, 8 up, 8 in 2024-04-27T01:17:55.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:55.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:55 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:55.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:55 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-27T01:17:55.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:55 smithi162 ceph-mon[26714]: osdmap e377: 8 total, 8 up, 8 in 2024-04-27T01:17:55.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:55.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:55 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:56.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:56.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:56 smithi181 ceph-mon[30237]: osdmap e378: 8 total, 8 up, 8 in 2024-04-27T01:17:56.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:56.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:56.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-53"}]: dispatch 2024-04-27T01:17:56.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-53"}]: dispatch 2024-04-27T01:17:56.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:56 smithi181 ceph-mon[30237]: pgmap v530: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-27T01:17:56.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[37680]: osdmap e378: 8 total, 8 up, 8 in 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-53"}]: dispatch 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-53"}]: dispatch 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[37680]: pgmap v530: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[26714]: osdmap e378: 8 total, 8 up, 8 in 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:56.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-53"}]: dispatch 2024-04-27T01:17:56.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-53"}]: dispatch 2024-04-27T01:17:56.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:56 smithi162 ceph-mon[26714]: pgmap v530: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-27T01:17:57.510 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:57 smithi181 ceph-mon[30237]: osdmap e379: 8 total, 8 up, 8 in 2024-04-27T01:17:57.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:57 smithi162 ceph-mon[37680]: osdmap e379: 8 total, 8 up, 8 in 2024-04-27T01:17:57.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:57 smithi162 ceph-mon[26714]: osdmap e379: 8 total, 8 up, 8 in 2024-04-27T01:17:58.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:58 smithi181 ceph-mon[30237]: osdmap e380: 8 total, 8 up, 8 in 2024-04-27T01:17:58.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:58.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:58.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:58 smithi181 ceph-mon[30237]: pgmap v533: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:17:58.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:58 smithi162 ceph-mon[37680]: osdmap e380: 8 total, 8 up, 8 in 2024-04-27T01:17:58.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:58.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:58.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:58 smithi162 ceph-mon[37680]: pgmap v533: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:17:58.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:58 smithi162 ceph-mon[26714]: osdmap e380: 8 total, 8 up, 8 in 2024-04-27T01:17:58.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:58.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:17:58.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:58 smithi162 ceph-mon[26714]: pgmap v533: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:17:59.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:59 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:59 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:59 smithi181 ceph-mon[30237]: osdmap e381: 8 total, 8 up, 8 in 2024-04-27T01:17:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:17:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:59 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-55"}]: dispatch 2024-04-27T01:17:59.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:17:59 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-55"}]: dispatch 2024-04-27T01:17:59.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:59.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[37680]: osdmap e381: 8 total, 8 up, 8 in 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-55"}]: dispatch 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-55"}]: dispatch 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[26714]: osdmap e381: 8 total, 8 up, 8 in 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:17:59.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:59.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:17:59.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-55"}]: dispatch 2024-04-27T01:17:59.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-55"}]: dispatch 2024-04-27T01:18:00.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:17:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:17:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:18:00.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:00 smithi181 ceph-mon[30237]: osdmap e382: 8 total, 8 up, 8 in 2024-04-27T01:18:00.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:00 smithi181 ceph-mon[30237]: pgmap v536: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:18:00.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:00 smithi162 ceph-mon[37680]: osdmap e382: 8 total, 8 up, 8 in 2024-04-27T01:18:00.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:00 smithi162 ceph-mon[37680]: pgmap v536: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:18:00.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:00 smithi162 ceph-mon[26714]: osdmap e382: 8 total, 8 up, 8 in 2024-04-27T01:18:00.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:00 smithi162 ceph-mon[26714]: pgmap v536: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:18:01.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:01 smithi181 ceph-mon[30237]: osdmap e383: 8 total, 8 up, 8 in 2024-04-27T01:18:01.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:01.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:01 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:01 smithi162 ceph-mon[37680]: osdmap e383: 8 total, 8 up, 8 in 2024-04-27T01:18:01.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:01.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:01 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:01.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:01 smithi162 ceph-mon[26714]: osdmap e383: 8 total, 8 up, 8 in 2024-04-27T01:18:01.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:01.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:01 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:02.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:18:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:18:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:02 smithi181 ceph-mon[30237]: osdmap e384: 8 total, 8 up, 8 in 2024-04-27T01:18:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:02 smithi181 ceph-mon[30237]: pgmap v539: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-57"}]: dispatch 2024-04-27T01:18:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-57"}]: dispatch 2024-04-27T01:18:02.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:02 smithi181 ceph-mon[30237]: osdmap e385: 8 total, 8 up, 8 in 2024-04-27T01:18:02.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:02.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[37680]: osdmap e384: 8 total, 8 up, 8 in 2024-04-27T01:18:02.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[37680]: pgmap v539: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-57"}]: dispatch 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-57"}]: dispatch 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[37680]: osdmap e385: 8 total, 8 up, 8 in 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[26714]: osdmap e384: 8 total, 8 up, 8 in 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[26714]: pgmap v539: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:02.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-57"}]: dispatch 2024-04-27T01:18:02.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-57"}]: dispatch 2024-04-27T01:18:02.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:02 smithi162 ceph-mon[26714]: osdmap e385: 8 total, 8 up, 8 in 2024-04-27T01:18:03.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:03.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:03 smithi181 ceph-mon[30237]: osdmap e386: 8 total, 8 up, 8 in 2024-04-27T01:18:03.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:03.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:03.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:03.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:03 smithi162 ceph-mon[37680]: osdmap e386: 8 total, 8 up, 8 in 2024-04-27T01:18:03.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:03.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:03 smithi162 ceph-mon[26714]: osdmap e386: 8 total, 8 up, 8 in 2024-04-27T01:18:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:03.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:04 smithi181 ceph-mon[30237]: pgmap v542: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:18:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:04 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:04 smithi181 ceph-mon[30237]: osdmap e387: 8 total, 8 up, 8 in 2024-04-27T01:18:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-59"}]: dispatch 2024-04-27T01:18:04.760 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-59"}]: dispatch 2024-04-27T01:18:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[37680]: pgmap v542: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[37680]: osdmap e387: 8 total, 8 up, 8 in 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-59"}]: dispatch 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-59"}]: dispatch 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[26714]: pgmap v542: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[26714]: osdmap e387: 8 total, 8 up, 8 in 2024-04-27T01:18:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-59"}]: dispatch 2024-04-27T01:18:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-59"}]: dispatch 2024-04-27T01:18:06.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:06 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:18:06.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:06 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:06.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:06 smithi181 ceph-mon[30237]: osdmap e388: 8 total, 8 up, 8 in 2024-04-27T01:18:06.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:06 smithi181 ceph-mon[30237]: pgmap v545: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:18:06.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:06 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:18:06.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:06 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:06.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:06 smithi162 ceph-mon[37680]: osdmap e388: 8 total, 8 up, 8 in 2024-04-27T01:18:06.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:06 smithi162 ceph-mon[37680]: pgmap v545: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:18:06.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:06 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:18:06.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:06 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:06.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:06 smithi162 ceph-mon[26714]: osdmap e388: 8 total, 8 up, 8 in 2024-04-27T01:18:06.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:06 smithi162 ceph-mon[26714]: pgmap v545: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:18:07.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:07 smithi181 ceph-mon[30237]: osdmap e389: 8 total, 8 up, 8 in 2024-04-27T01:18:07.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:07.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:07 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:07.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:07 smithi162 ceph-mon[37680]: osdmap e389: 8 total, 8 up, 8 in 2024-04-27T01:18:07.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:07.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:07 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:07.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:07 smithi162 ceph-mon[26714]: osdmap e389: 8 total, 8 up, 8 in 2024-04-27T01:18:07.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:07.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:07 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:08.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:08.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:08 smithi181 ceph-mon[30237]: osdmap e390: 8 total, 8 up, 8 in 2024-04-27T01:18:08.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:08 smithi181 ceph-mon[30237]: pgmap v548: 228 pgs: 4 creating+activating, 21 creating+peering, 5 unknown, 198 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:18:08.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[37680]: osdmap e390: 8 total, 8 up, 8 in 2024-04-27T01:18:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[37680]: pgmap v548: 228 pgs: 4 creating+activating, 21 creating+peering, 5 unknown, 198 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:18:08.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:08.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[26714]: osdmap e390: 8 total, 8 up, 8 in 2024-04-27T01:18:08.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:08.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:08.588 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:08 smithi162 ceph-mon[26714]: pgmap v548: 228 pgs: 4 creating+activating, 21 creating+peering, 5 unknown, 198 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:18:09.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:18:09.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:09 smithi181 ceph-mon[30237]: osdmap e391: 8 total, 8 up, 8 in 2024-04-27T01:18:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-61"}]: dispatch 2024-04-27T01:18:09.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-61"}]: dispatch 2024-04-27T01:18:09.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:18:09.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[37680]: osdmap e391: 8 total, 8 up, 8 in 2024-04-27T01:18:09.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:09.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:09.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-61"}]: dispatch 2024-04-27T01:18:09.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-61"}]: dispatch 2024-04-27T01:18:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:18:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[26714]: osdmap e391: 8 total, 8 up, 8 in 2024-04-27T01:18:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-61"}]: dispatch 2024-04-27T01:18:09.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-61"}]: dispatch 2024-04-27T01:18:10.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:18:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:18:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:18:10.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:10 smithi181 ceph-mon[30237]: osdmap e392: 8 total, 8 up, 8 in 2024-04-27T01:18:10.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:10 smithi181 ceph-mon[30237]: pgmap v551: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:18:10.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:10 smithi162 ceph-mon[37680]: osdmap e392: 8 total, 8 up, 8 in 2024-04-27T01:18:10.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:10 smithi162 ceph-mon[37680]: pgmap v551: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:18:10.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:10 smithi162 ceph-mon[26714]: osdmap e392: 8 total, 8 up, 8 in 2024-04-27T01:18:10.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:10 smithi162 ceph-mon[26714]: pgmap v551: 196 pgs: 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:18:11.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:11 smithi181 ceph-mon[30237]: osdmap e393: 8 total, 8 up, 8 in 2024-04-27T01:18:11.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:11.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:11 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:11.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:11 smithi162 ceph-mon[37680]: osdmap e393: 8 total, 8 up, 8 in 2024-04-27T01:18:11.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:11.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:11 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:11.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:11 smithi162 ceph-mon[26714]: osdmap e393: 8 total, 8 up, 8 in 2024-04-27T01:18:11.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:11.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:11 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:12.488 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[26714]: pgmap v553: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:12.488 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:12.488 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[26714]: osdmap e394: 8 total, 8 up, 8 in 2024-04-27T01:18:12.488 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:12.488 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:12.488 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:12.488 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[37680]: pgmap v553: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[37680]: osdmap e394: 8 total, 8 up, 8 in 2024-04-27T01:18:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:12.489 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:12 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:12.508 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:18:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:18:12.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:12 smithi181 ceph-mon[30237]: pgmap v553: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:12.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:12.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:12 smithi181 ceph-mon[30237]: osdmap e394: 8 total, 8 up, 8 in 2024-04-27T01:18:12.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:12.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:12.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:12.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:12 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:13.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:13 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:18:13.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:13 smithi181 ceph-mon[30237]: osdmap e395: 8 total, 8 up, 8 in 2024-04-27T01:18:13.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:13.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:13 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:18:13.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:13 smithi162 ceph-mon[26714]: osdmap e395: 8 total, 8 up, 8 in 2024-04-27T01:18:13.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:13 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:18:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:13 smithi162 ceph-mon[37680]: osdmap e395: 8 total, 8 up, 8 in 2024-04-27T01:18:13.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:14.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:14 smithi181 ceph-mon[30237]: pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 510 B/s wr, 1 op/s 2024-04-27T01:18:14.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:14 smithi181 ceph-mon[30237]: osdmap e396: 8 total, 8 up, 8 in 2024-04-27T01:18:14.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:14 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:18:14.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:14 smithi162 ceph-mon[26714]: pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 510 B/s wr, 1 op/s 2024-04-27T01:18:14.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:14 smithi162 ceph-mon[26714]: osdmap e396: 8 total, 8 up, 8 in 2024-04-27T01:18:14.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:14 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:18:14.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:14 smithi162 ceph-mon[37680]: pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 510 B/s wr, 1 op/s 2024-04-27T01:18:14.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:14 smithi162 ceph-mon[37680]: osdmap e396: 8 total, 8 up, 8 in 2024-04-27T01:18:14.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:14 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:18:15.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:15 smithi181 ceph-mon[30237]: osdmap e397: 8 total, 8 up, 8 in 2024-04-27T01:18:15.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:15 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:15.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:15 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:18:15.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:15 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:18:15.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:15 smithi162 ceph-mon[26714]: osdmap e397: 8 total, 8 up, 8 in 2024-04-27T01:18:15.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:15 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:15.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:15 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:18:15.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:15 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:18:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:15 smithi162 ceph-mon[37680]: osdmap e397: 8 total, 8 up, 8 in 2024-04-27T01:18:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:15 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:18:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:15 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:18:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:15 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:18:16.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:16 smithi181 ceph-mon[30237]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:18:16.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:16 smithi181 ceph-mon[30237]: osdmap e398: 8 total, 8 up, 8 in 2024-04-27T01:18:16.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:16 smithi162 ceph-mon[37680]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:18:16.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:16 smithi162 ceph-mon[37680]: osdmap e398: 8 total, 8 up, 8 in 2024-04-27T01:18:16.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:16 smithi162 ceph-mon[26714]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:18:16.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:16 smithi162 ceph-mon[26714]: osdmap e398: 8 total, 8 up, 8 in 2024-04-27T01:18:17.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:17 smithi162 ceph-mon[37680]: osdmap e399: 8 total, 8 up, 8 in 2024-04-27T01:18:17.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:17 smithi162 ceph-mon[26714]: osdmap e399: 8 total, 8 up, 8 in 2024-04-27T01:18:17.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:17 smithi181 ceph-mon[30237]: osdmap e399: 8 total, 8 up, 8 in 2024-04-27T01:18:18.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:18 smithi162 ceph-mon[37680]: pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-27T01:18:18.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:18 smithi162 ceph-mon[26714]: pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-27T01:18:18.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:18 smithi181 ceph-mon[30237]: pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-27T01:18:19.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:19 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:19.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:19 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:19.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:19 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:20.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:18:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:18:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:18:20.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:20 smithi162 ceph-mon[26714]: pgmap v563: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-04-27T01:18:20.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:20 smithi162 ceph-mon[37680]: pgmap v563: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-04-27T01:18:20.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:20.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:20 smithi181 ceph-mon[30237]: pgmap v563: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-04-27T01:18:20.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:22.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:22 smithi162 ceph-mon[37680]: pgmap v564: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-27T01:18:22.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:22 smithi162 ceph-mon[26714]: pgmap v564: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-27T01:18:22.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:18:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:18:22.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:22 smithi181 ceph-mon[30237]: pgmap v564: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-27T01:18:23.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:23.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:23.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:24.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:24 smithi162 ceph-mon[26714]: pgmap v565: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-27T01:18:24.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:24 smithi162 ceph-mon[26714]: osdmap e400: 8 total, 8 up, 8 in 2024-04-27T01:18:24.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:24 smithi162 ceph-mon[37680]: pgmap v565: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-27T01:18:24.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:24 smithi162 ceph-mon[37680]: osdmap e400: 8 total, 8 up, 8 in 2024-04-27T01:18:24.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:24 smithi181 ceph-mon[30237]: pgmap v565: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-04-27T01:18:24.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:24 smithi181 ceph-mon[30237]: osdmap e400: 8 total, 8 up, 8 in 2024-04-27T01:18:26.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:26 smithi162 ceph-mon[26714]: pgmap v567: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T01:18:26.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:26 smithi162 ceph-mon[37680]: pgmap v567: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T01:18:26.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:26 smithi181 ceph-mon[30237]: pgmap v567: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T01:18:27.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:27 smithi162 ceph-mon[37680]: osdmap e401: 8 total, 8 up, 8 in 2024-04-27T01:18:27.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:27 smithi162 ceph-mon[26714]: osdmap e401: 8 total, 8 up, 8 in 2024-04-27T01:18:27.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:27 smithi181 ceph-mon[30237]: osdmap e401: 8 total, 8 up, 8 in 2024-04-27T01:18:28.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:28 smithi162 ceph-mon[37680]: pgmap v569: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:28.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:28 smithi162 ceph-mon[26714]: pgmap v569: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:28.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:28 smithi181 ceph-mon[30237]: pgmap v569: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:30.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:18:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:18:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:18:30.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:30 smithi162 ceph-mon[37680]: pgmap v570: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:30.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:30 smithi162 ceph-mon[26714]: pgmap v570: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:30.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:30 smithi181 ceph-mon[30237]: pgmap v570: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:32.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:32 smithi162 ceph-mon[37680]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:32.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:32 smithi162 ceph-mon[26714]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:32.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:32 smithi181 ceph-mon[30237]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:32.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:18:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:18:33.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:33.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:33.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:34.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:34 smithi162 ceph-mon[37680]: pgmap v572: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:34.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:34 smithi162 ceph-mon[37680]: osdmap e402: 8 total, 8 up, 8 in 2024-04-27T01:18:34.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:34 smithi162 ceph-mon[26714]: pgmap v572: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:34.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:34 smithi162 ceph-mon[26714]: osdmap e402: 8 total, 8 up, 8 in 2024-04-27T01:18:34.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:34 smithi181 ceph-mon[30237]: pgmap v572: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:34.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:34 smithi181 ceph-mon[30237]: osdmap e402: 8 total, 8 up, 8 in 2024-04-27T01:18:35.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:35 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:35.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:35 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:35.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:35 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:36.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:36 smithi181 ceph-mon[30237]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:36.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:36 smithi162 ceph-mon[37680]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:36.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:36 smithi162 ceph-mon[26714]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:37.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:37 smithi181 ceph-mon[30237]: osdmap e403: 8 total, 8 up, 8 in 2024-04-27T01:18:37.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:37 smithi162 ceph-mon[26714]: osdmap e403: 8 total, 8 up, 8 in 2024-04-27T01:18:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:37 smithi162 ceph-mon[37680]: osdmap e403: 8 total, 8 up, 8 in 2024-04-27T01:18:38.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:38 smithi181 ceph-mon[30237]: pgmap v576: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:18:38.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:38 smithi162 ceph-mon[37680]: pgmap v576: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:18:38.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:38 smithi162 ceph-mon[26714]: pgmap v576: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:18:40.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:18:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:18:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:18:40.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:40 smithi181 ceph-mon[30237]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:18:40.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:40 smithi162 ceph-mon[37680]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:18:40.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:40 smithi162 ceph-mon[26714]: pgmap v577: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:18:42.639 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:42 smithi162 ceph-mon[37680]: pgmap v578: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-27T01:18:42.639 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:42 smithi162 ceph-mon[26714]: pgmap v578: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-27T01:18:42.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:42 smithi181 ceph-mon[30237]: pgmap v578: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-27T01:18:42.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:18:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:18:43.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:43.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:43.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:44.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:44 smithi181 ceph-mon[30237]: pgmap v579: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:18:44.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:44 smithi181 ceph-mon[30237]: osdmap e404: 8 total, 8 up, 8 in 2024-04-27T01:18:44.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:44 smithi162 ceph-mon[37680]: pgmap v579: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:18:44.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:44 smithi162 ceph-mon[37680]: osdmap e404: 8 total, 8 up, 8 in 2024-04-27T01:18:44.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:44 smithi162 ceph-mon[26714]: pgmap v579: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:18:44.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:44 smithi162 ceph-mon[26714]: osdmap e404: 8 total, 8 up, 8 in 2024-04-27T01:18:46.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:46 smithi181 ceph-mon[30237]: pgmap v581: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:18:46.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:46.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:46 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:46.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-63"}]: dispatch 2024-04-27T01:18:46.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:46 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-63"}]: dispatch 2024-04-27T01:18:46.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:46 smithi162 ceph-mon[26714]: pgmap v581: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:18:46.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:46.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:46 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:46.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-63"}]: dispatch 2024-04-27T01:18:46.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:46 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-63"}]: dispatch 2024-04-27T01:18:46.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:46 smithi162 ceph-mon[37680]: pgmap v581: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:18:46.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:46.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:46 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:18:46.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-63"}]: dispatch 2024-04-27T01:18:46.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:46 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-63"}]: dispatch 2024-04-27T01:18:48.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:48 smithi181 ceph-mon[30237]: pgmap v582: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:48.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:48 smithi181 ceph-mon[30237]: osdmap e405: 8 total, 8 up, 8 in 2024-04-27T01:18:48.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:48 smithi162 ceph-mon[37680]: pgmap v582: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:48.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:48 smithi162 ceph-mon[37680]: osdmap e405: 8 total, 8 up, 8 in 2024-04-27T01:18:48.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:48 smithi162 ceph-mon[26714]: pgmap v582: 228 pgs: 228 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:18:48.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:48 smithi162 ceph-mon[26714]: osdmap e405: 8 total, 8 up, 8 in 2024-04-27T01:18:49.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:49 smithi162 ceph-mon[26714]: osdmap e406: 8 total, 8 up, 8 in 2024-04-27T01:18:49.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:49.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:49 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:49.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:49 smithi162 ceph-mon[37680]: osdmap e406: 8 total, 8 up, 8 in 2024-04-27T01:18:49.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:49.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:49 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:49.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:49 smithi181 ceph-mon[30237]: osdmap e406: 8 total, 8 up, 8 in 2024-04-27T01:18:49.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:49.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:49 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:18:50.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:18:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:18:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:18:50.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:50 smithi181 ceph-mon[30237]: pgmap v585: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:50.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:50 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:50.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:50 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:50.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:50 smithi181 ceph-mon[30237]: osdmap e407: 8 total, 8 up, 8 in 2024-04-27T01:18:50.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:50.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:50.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:50 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:50.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[26714]: pgmap v585: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[26714]: osdmap e407: 8 total, 8 up, 8 in 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[37680]: pgmap v585: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[37680]: osdmap e407: 8 total, 8 up, 8 in 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:18:50.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:50.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:18:50.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:18:51.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:51 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:18:51.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:51 smithi181 ceph-mon[30237]: osdmap e408: 8 total, 8 up, 8 in 2024-04-27T01:18:51.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:51 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:18:51.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:51 smithi162 ceph-mon[26714]: osdmap e408: 8 total, 8 up, 8 in 2024-04-27T01:18:51.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:51 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:18:51.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:51 smithi162 ceph-mon[37680]: osdmap e408: 8 total, 8 up, 8 in 2024-04-27T01:18:52.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:52 smithi181 ceph-mon[30237]: pgmap v588: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:52.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:52 smithi181 ceph-mon[30237]: osdmap e409: 8 total, 8 up, 8 in 2024-04-27T01:18:52.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:18:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:18:52.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:52 smithi162 ceph-mon[26714]: pgmap v588: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:52.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:52 smithi162 ceph-mon[26714]: osdmap e409: 8 total, 8 up, 8 in 2024-04-27T01:18:52.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:52 smithi162 ceph-mon[37680]: pgmap v588: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:18:52.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:52 smithi162 ceph-mon[37680]: osdmap e409: 8 total, 8 up, 8 in 2024-04-27T01:18:53.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:53 smithi181 ceph-mon[30237]: osdmap e410: 8 total, 8 up, 8 in 2024-04-27T01:18:53.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:53.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:53 smithi162 ceph-mon[26714]: osdmap e410: 8 total, 8 up, 8 in 2024-04-27T01:18:53.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:53.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:53 smithi162 ceph-mon[37680]: osdmap e410: 8 total, 8 up, 8 in 2024-04-27T01:18:53.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:18:54.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:54 smithi181 ceph-mon[30237]: pgmap v591: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-27T01:18:54.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:54 smithi181 ceph-mon[30237]: osdmap e411: 8 total, 8 up, 8 in 2024-04-27T01:18:54.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:54 smithi162 ceph-mon[26714]: pgmap v591: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-27T01:18:54.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:54 smithi162 ceph-mon[26714]: osdmap e411: 8 total, 8 up, 8 in 2024-04-27T01:18:54.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:54 smithi162 ceph-mon[37680]: pgmap v591: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-27T01:18:54.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:54 smithi162 ceph-mon[37680]: osdmap e411: 8 total, 8 up, 8 in 2024-04-27T01:18:56.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:56 smithi181 ceph-mon[30237]: pgmap v593: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-27T01:18:56.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:56 smithi162 ceph-mon[26714]: pgmap v593: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-27T01:18:56.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:56 smithi162 ceph-mon[37680]: pgmap v593: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-27T01:18:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:58 smithi181 ceph-mon[30237]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-27T01:18:58.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:58 smithi162 ceph-mon[26714]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-27T01:18:58.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:58 smithi162 ceph-mon[37680]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-27T01:18:59.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:18:59 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:59.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:18:59 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:18:59.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:18:59 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:00.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:18:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:18:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:19:00.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:00 smithi181 ceph-mon[30237]: pgmap v595: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-27T01:19:00.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:00 smithi162 ceph-mon[26714]: pgmap v595: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-27T01:19:00.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:00 smithi162 ceph-mon[37680]: pgmap v595: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-04-27T01:19:02.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:19:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:19:02.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:02 smithi181 ceph-mon[30237]: pgmap v596: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 231 B/s wr, 2 op/s 2024-04-27T01:19:02.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:02 smithi162 ceph-mon[26714]: pgmap v596: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 231 B/s wr, 2 op/s 2024-04-27T01:19:02.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:02 smithi162 ceph-mon[37680]: pgmap v596: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 231 B/s wr, 2 op/s 2024-04-27T01:19:03.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:03.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:03.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:04 smithi181 ceph-mon[30237]: pgmap v597: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-27T01:19:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-65"}]: dispatch 2024-04-27T01:19:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-65"}]: dispatch 2024-04-27T01:19:04.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:04 smithi181 ceph-mon[30237]: osdmap e412: 8 total, 8 up, 8 in 2024-04-27T01:19:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[37680]: pgmap v597: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-27T01:19:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-65"}]: dispatch 2024-04-27T01:19:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-65"}]: dispatch 2024-04-27T01:19:04.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[37680]: osdmap e412: 8 total, 8 up, 8 in 2024-04-27T01:19:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[26714]: pgmap v597: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-27T01:19:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-65"}]: dispatch 2024-04-27T01:19:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-65"}]: dispatch 2024-04-27T01:19:04.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:04 smithi162 ceph-mon[26714]: osdmap e412: 8 total, 8 up, 8 in 2024-04-27T01:19:06.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:06 smithi181 ceph-mon[30237]: osdmap e413: 8 total, 8 up, 8 in 2024-04-27T01:19:06.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:06.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:06 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:06.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:06 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:06.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:06 smithi181 ceph-mon[30237]: pgmap v600: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:06.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:06 smithi162 ceph-mon[37680]: osdmap e413: 8 total, 8 up, 8 in 2024-04-27T01:19:06.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:06.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:06 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:06.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:06 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:06.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:06 smithi162 ceph-mon[37680]: pgmap v600: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:06.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:06 smithi162 ceph-mon[26714]: osdmap e413: 8 total, 8 up, 8 in 2024-04-27T01:19:06.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:06.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:06 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:06.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:06 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:06.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:06 smithi162 ceph-mon[26714]: pgmap v600: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:07.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:07 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:07.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:07 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:07.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:07 smithi181 ceph-mon[30237]: osdmap e414: 8 total, 8 up, 8 in 2024-04-27T01:19:07.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:07.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:07 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:07.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:07 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:07.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:07 smithi162 ceph-mon[37680]: osdmap e414: 8 total, 8 up, 8 in 2024-04-27T01:19:07.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:07.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:07 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:07.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:07 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:07.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:07 smithi162 ceph-mon[26714]: osdmap e414: 8 total, 8 up, 8 in 2024-04-27T01:19:07.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:08.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:08 smithi181 ceph-mon[30237]: osdmap e415: 8 total, 8 up, 8 in 2024-04-27T01:19:08.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:08.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:08.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-67"}]: dispatch 2024-04-27T01:19:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-67"}]: dispatch 2024-04-27T01:19:08.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:08 smithi181 ceph-mon[30237]: pgmap v603: 228 pgs: 4 unknown, 224 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:08.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[26714]: osdmap e415: 8 total, 8 up, 8 in 2024-04-27T01:19:08.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:08.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:08.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-67"}]: dispatch 2024-04-27T01:19:08.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-67"}]: dispatch 2024-04-27T01:19:08.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[26714]: pgmap v603: 228 pgs: 4 unknown, 224 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[37680]: osdmap e415: 8 total, 8 up, 8 in 2024-04-27T01:19:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-67"}]: dispatch 2024-04-27T01:19:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-67"}]: dispatch 2024-04-27T01:19:08.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:08 smithi162 ceph-mon[37680]: pgmap v603: 228 pgs: 4 unknown, 224 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:09.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:09 smithi181 ceph-mon[30237]: osdmap e416: 8 total, 8 up, 8 in 2024-04-27T01:19:09.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:09 smithi162 ceph-mon[37680]: osdmap e416: 8 total, 8 up, 8 in 2024-04-27T01:19:09.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:09 smithi162 ceph-mon[26714]: osdmap e416: 8 total, 8 up, 8 in 2024-04-27T01:19:10.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:19:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:19:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:19:10.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:10 smithi181 ceph-mon[30237]: osdmap e417: 8 total, 8 up, 8 in 2024-04-27T01:19:10.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:10.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:10.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:10 smithi181 ceph-mon[30237]: pgmap v606: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:19:10.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:10 smithi162 ceph-mon[37680]: osdmap e417: 8 total, 8 up, 8 in 2024-04-27T01:19:10.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:10.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:10.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:10 smithi162 ceph-mon[37680]: pgmap v606: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:19:10.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:10 smithi162 ceph-mon[26714]: osdmap e417: 8 total, 8 up, 8 in 2024-04-27T01:19:10.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:10.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:10.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:10 smithi162 ceph-mon[26714]: pgmap v606: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:19:11.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:11 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:11.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:11 smithi181 ceph-mon[30237]: osdmap e418: 8 total, 8 up, 8 in 2024-04-27T01:19:11.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:11.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:11.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:11 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:11.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-69"}]: dispatch 2024-04-27T01:19:11.509 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:11 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-69"}]: dispatch 2024-04-27T01:19:11.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:11.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[37680]: osdmap e418: 8 total, 8 up, 8 in 2024-04-27T01:19:11.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:11.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:11.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:11.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-69"}]: dispatch 2024-04-27T01:19:11.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-69"}]: dispatch 2024-04-27T01:19:11.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:11.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[26714]: osdmap e418: 8 total, 8 up, 8 in 2024-04-27T01:19:11.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:11.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:11.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:11.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-69"}]: dispatch 2024-04-27T01:19:11.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:11 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-69"}]: dispatch 2024-04-27T01:19:12.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:12 smithi162 ceph-mon[37680]: osdmap e419: 8 total, 8 up, 8 in 2024-04-27T01:19:12.487 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:12 smithi162 ceph-mon[37680]: pgmap v609: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:12.488 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:12 smithi162 ceph-mon[26714]: osdmap e419: 8 total, 8 up, 8 in 2024-04-27T01:19:12.488 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:12 smithi162 ceph-mon[26714]: pgmap v609: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:12.504 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:12 smithi181 ceph-mon[30237]: osdmap e419: 8 total, 8 up, 8 in 2024-04-27T01:19:12.504 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:12 smithi181 ceph-mon[30237]: pgmap v609: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:12.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:19:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:19:13.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:13 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:13.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:13 smithi181 ceph-mon[30237]: osdmap e420: 8 total, 8 up, 8 in 2024-04-27T01:19:13.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:13.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:13 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:13.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:13.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:13 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:13.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:13 smithi162 ceph-mon[37680]: osdmap e420: 8 total, 8 up, 8 in 2024-04-27T01:19:13.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:13.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:13 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:13.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:13.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:13 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:13.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:13 smithi162 ceph-mon[26714]: osdmap e420: 8 total, 8 up, 8 in 2024-04-27T01:19:13.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:13.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:13 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:13.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:14.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:14 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:14.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:14 smithi181 ceph-mon[30237]: osdmap e421: 8 total, 8 up, 8 in 2024-04-27T01:19:14.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:14.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:19:14.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:14 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:19:14.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:14 smithi181 ceph-mon[30237]: pgmap v612: 228 pgs: 13 creating+peering, 19 unknown, 196 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:19:14.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:14.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[37680]: osdmap e421: 8 total, 8 up, 8 in 2024-04-27T01:19:14.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:14.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:19:14.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:19:14.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[37680]: pgmap v612: 228 pgs: 13 creating+peering, 19 unknown, 196 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:19:14.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:14.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[26714]: osdmap e421: 8 total, 8 up, 8 in 2024-04-27T01:19:14.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:14.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:19:14.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:19:14.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:14 smithi162 ceph-mon[26714]: pgmap v612: 228 pgs: 13 creating+peering, 19 unknown, 196 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:19:15.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:15 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:19:15.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:15 smithi181 ceph-mon[30237]: osdmap e422: 8 total, 8 up, 8 in 2024-04-27T01:19:15.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:15 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:15.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:15 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:15.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:15 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:19:15.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:15 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:19:15.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:19:15.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[26714]: osdmap e422: 8 total, 8 up, 8 in 2024-04-27T01:19:15.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:15.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:15.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:19:15.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:19:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:19:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[37680]: osdmap e422: 8 total, 8 up, 8 in 2024-04-27T01:19:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:19:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:19:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:19:15.587 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:15 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:19:16.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:16 smithi181 ceph-mon[30237]: osdmap e423: 8 total, 8 up, 8 in 2024-04-27T01:19:16.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:16 smithi181 ceph-mon[30237]: pgmap v615: 228 pgs: 13 creating+peering, 19 unknown, 196 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:19:16.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:16 smithi162 ceph-mon[26714]: osdmap e423: 8 total, 8 up, 8 in 2024-04-27T01:19:16.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:16 smithi162 ceph-mon[26714]: pgmap v615: 228 pgs: 13 creating+peering, 19 unknown, 196 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:19:16.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:16 smithi162 ceph-mon[37680]: osdmap e423: 8 total, 8 up, 8 in 2024-04-27T01:19:16.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:16 smithi162 ceph-mon[37680]: pgmap v615: 228 pgs: 13 creating+peering, 19 unknown, 196 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:19:18.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:18 smithi162 ceph-mon[26714]: pgmap v616: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 5 op/s 2024-04-27T01:19:18.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:18 smithi162 ceph-mon[37680]: pgmap v616: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 5 op/s 2024-04-27T01:19:18.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:18 smithi181 ceph-mon[30237]: pgmap v616: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 5 op/s 2024-04-27T01:19:20.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:19:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:19:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:19:20.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:20 smithi162 ceph-mon[37680]: pgmap v617: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-04-27T01:19:20.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:20.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:20 smithi162 ceph-mon[26714]: pgmap v617: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-04-27T01:19:20.587 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:20.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:20 smithi181 ceph-mon[30237]: pgmap v617: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-04-27T01:19:20.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:22.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:22 smithi162 ceph-mon[37680]: pgmap v618: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-27T01:19:22.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:22 smithi162 ceph-mon[26714]: pgmap v618: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-27T01:19:22.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:19:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:19:22.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:22 smithi181 ceph-mon[30237]: pgmap v618: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-27T01:19:23.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:23.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:23.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:24.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:24 smithi162 ceph-mon[37680]: pgmap v619: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-27T01:19:24.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:24 smithi162 ceph-mon[26714]: pgmap v619: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-27T01:19:24.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:24 smithi181 ceph-mon[30237]: pgmap v619: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-27T01:19:26.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:26 smithi162 ceph-mon[37680]: pgmap v620: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-27T01:19:26.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:26 smithi162 ceph-mon[26714]: pgmap v620: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-27T01:19:26.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:26 smithi181 ceph-mon[30237]: pgmap v620: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-27T01:19:28.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:28 smithi162 ceph-mon[37680]: pgmap v621: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-27T01:19:28.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:28 smithi162 ceph-mon[26714]: pgmap v621: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-27T01:19:28.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:28 smithi181 ceph-mon[30237]: pgmap v621: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-27T01:19:30.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:19:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:19:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:19:30.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:30 smithi162 ceph-mon[37680]: pgmap v622: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:30.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:30 smithi162 ceph-mon[26714]: pgmap v622: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:30.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:30 smithi181 ceph-mon[30237]: pgmap v622: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:32.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:32 smithi162 ceph-mon[37680]: pgmap v623: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:32.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:32 smithi162 ceph-mon[26714]: pgmap v623: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:32.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:32 smithi181 ceph-mon[30237]: pgmap v623: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:32.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:19:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:19:33.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:33.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:33.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:34.586 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:34 smithi162 ceph-mon[37680]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:34.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:34 smithi162 ceph-mon[26714]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:34.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:34 smithi181 ceph-mon[30237]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:35.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:35 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:35.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:35.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:35 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:35.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-71"}]: dispatch 2024-04-27T01:19:35.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:35 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-71"}]: dispatch 2024-04-27T01:19:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:35 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:35 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-71"}]: dispatch 2024-04-27T01:19:35.838 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:35 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-71"}]: dispatch 2024-04-27T01:19:35.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:35 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:35.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:35.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:35 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:35.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-71"}]: dispatch 2024-04-27T01:19:35.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:35 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-71"}]: dispatch 2024-04-27T01:19:36.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:36 smithi181 ceph-mon[30237]: pgmap v625: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:36.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:36 smithi181 ceph-mon[30237]: osdmap e424: 8 total, 8 up, 8 in 2024-04-27T01:19:36.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:36 smithi162 ceph-mon[37680]: pgmap v625: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:36.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:36 smithi162 ceph-mon[37680]: osdmap e424: 8 total, 8 up, 8 in 2024-04-27T01:19:36.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:36 smithi162 ceph-mon[26714]: pgmap v625: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:19:36.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:36 smithi162 ceph-mon[26714]: osdmap e424: 8 total, 8 up, 8 in 2024-04-27T01:19:37.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:37 smithi181 ceph-mon[30237]: osdmap e425: 8 total, 8 up, 8 in 2024-04-27T01:19:37.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:37.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:37 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:37 smithi162 ceph-mon[37680]: osdmap e425: 8 total, 8 up, 8 in 2024-04-27T01:19:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:37.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:37 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:37 smithi162 ceph-mon[26714]: osdmap e425: 8 total, 8 up, 8 in 2024-04-27T01:19:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:37.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:37 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:38.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:38 smithi181 ceph-mon[30237]: pgmap v628: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:38.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:38 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:38.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:38 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:38.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:38 smithi181 ceph-mon[30237]: osdmap e426: 8 total, 8 up, 8 in 2024-04-27T01:19:38.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:38.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:38 smithi162 ceph-mon[37680]: pgmap v628: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:38.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:38 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:38.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:38 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:38.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:38 smithi162 ceph-mon[37680]: osdmap e426: 8 total, 8 up, 8 in 2024-04-27T01:19:38.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:38 smithi162 ceph-mon[26714]: pgmap v628: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:38 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:38 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:38 smithi162 ceph-mon[26714]: osdmap e426: 8 total, 8 up, 8 in 2024-04-27T01:19:38.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:39.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:39 smithi162 ceph-mon[37680]: osdmap e427: 8 total, 8 up, 8 in 2024-04-27T01:19:39.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:39 smithi162 ceph-mon[26714]: osdmap e427: 8 total, 8 up, 8 in 2024-04-27T01:19:39.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:39 smithi181 ceph-mon[30237]: osdmap e427: 8 total, 8 up, 8 in 2024-04-27T01:19:40.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:19:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:19:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:19:40.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:40 smithi181 ceph-mon[30237]: pgmap v631: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:40.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:40 smithi181 ceph-mon[30237]: osdmap e428: 8 total, 8 up, 8 in 2024-04-27T01:19:40.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:40 smithi162 ceph-mon[37680]: pgmap v631: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:40.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:40 smithi162 ceph-mon[37680]: osdmap e428: 8 total, 8 up, 8 in 2024-04-27T01:19:40.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:40 smithi162 ceph-mon[26714]: pgmap v631: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:40.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:40 smithi162 ceph-mon[26714]: osdmap e428: 8 total, 8 up, 8 in 2024-04-27T01:19:42.638 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:42 smithi162 ceph-mon[37680]: pgmap v633: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-27T01:19:42.639 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:42 smithi162 ceph-mon[26714]: pgmap v633: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-27T01:19:42.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:19:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:19:42.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:42 smithi181 ceph-mon[30237]: pgmap v633: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-27T01:19:43.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:43.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:43.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:44.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:44 smithi181 ceph-mon[30237]: pgmap v634: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T01:19:44.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:44 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:44.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:44 smithi162 ceph-mon[26714]: pgmap v634: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T01:19:44.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:44 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:44.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:44 smithi162 ceph-mon[37680]: pgmap v634: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-27T01:19:44.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:44 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:46.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:46 smithi181 ceph-mon[30237]: pgmap v635: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 907 B/s wr, 3 op/s 2024-04-27T01:19:46.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:46 smithi162 ceph-mon[37680]: pgmap v635: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 907 B/s wr, 3 op/s 2024-04-27T01:19:46.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:46 smithi162 ceph-mon[26714]: pgmap v635: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 907 B/s wr, 3 op/s 2024-04-27T01:19:48.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:48 smithi181 ceph-mon[30237]: pgmap v636: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 805 B/s wr, 3 op/s 2024-04-27T01:19:48.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:48 smithi162 ceph-mon[37680]: pgmap v636: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 805 B/s wr, 3 op/s 2024-04-27T01:19:48.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:48 smithi162 ceph-mon[26714]: pgmap v636: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 805 B/s wr, 3 op/s 2024-04-27T01:19:50.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:19:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:19:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:19:50.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:50 smithi181 ceph-mon[30237]: pgmap v637: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-04-27T01:19:50.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:50.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:50 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:50.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-73"}]: dispatch 2024-04-27T01:19:50.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:50 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-73"}]: dispatch 2024-04-27T01:19:50.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:50.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[37680]: pgmap v637: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-04-27T01:19:50.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:50.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:50.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-73"}]: dispatch 2024-04-27T01:19:50.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-73"}]: dispatch 2024-04-27T01:19:50.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[26714]: pgmap v637: 228 pgs: 228 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-04-27T01:19:50.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:50.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:50.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-73"}]: dispatch 2024-04-27T01:19:50.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-73"}]: dispatch 2024-04-27T01:19:50.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:19:51.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:51 smithi181 ceph-mon[30237]: osdmap e429: 8 total, 8 up, 8 in 2024-04-27T01:19:51.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:51 smithi162 ceph-mon[37680]: osdmap e429: 8 total, 8 up, 8 in 2024-04-27T01:19:51.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:51 smithi162 ceph-mon[26714]: osdmap e429: 8 total, 8 up, 8 in 2024-04-27T01:19:52.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:19:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:19:52.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:52 smithi181 ceph-mon[30237]: pgmap v639: 196 pgs: 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 307 B/s wr, 1 op/s 2024-04-27T01:19:52.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:52 smithi181 ceph-mon[30237]: osdmap e430: 8 total, 8 up, 8 in 2024-04-27T01:19:52.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:52.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:52 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:52.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:52 smithi162 ceph-mon[37680]: pgmap v639: 196 pgs: 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 307 B/s wr, 1 op/s 2024-04-27T01:19:52.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:52 smithi162 ceph-mon[37680]: osdmap e430: 8 total, 8 up, 8 in 2024-04-27T01:19:52.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:52.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:52 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:52.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:52 smithi162 ceph-mon[26714]: pgmap v639: 196 pgs: 196 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 307 B/s wr, 1 op/s 2024-04-27T01:19:52.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:52 smithi162 ceph-mon[26714]: osdmap e430: 8 total, 8 up, 8 in 2024-04-27T01:19:52.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:52.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:52 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:53.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:53 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:53.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:53 smithi181 ceph-mon[30237]: osdmap e431: 8 total, 8 up, 8 in 2024-04-27T01:19:53.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:53.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:53.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:53 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:53.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:53 smithi162 ceph-mon[37680]: osdmap e431: 8 total, 8 up, 8 in 2024-04-27T01:19:53.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:53.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:53.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:53 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:53.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:53 smithi162 ceph-mon[26714]: osdmap e431: 8 total, 8 up, 8 in 2024-04-27T01:19:53.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:53.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:19:54.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:54 smithi181 ceph-mon[30237]: pgmap v642: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:54.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:54 smithi181 ceph-mon[30237]: osdmap e432: 8 total, 8 up, 8 in 2024-04-27T01:19:54.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:54 smithi162 ceph-mon[37680]: pgmap v642: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:54.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:54 smithi162 ceph-mon[37680]: osdmap e432: 8 total, 8 up, 8 in 2024-04-27T01:19:54.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:54 smithi162 ceph-mon[26714]: pgmap v642: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:19:54.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:54 smithi162 ceph-mon[26714]: osdmap e432: 8 total, 8 up, 8 in 2024-04-27T01:19:55.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:55 smithi181 ceph-mon[30237]: osdmap e433: 8 total, 8 up, 8 in 2024-04-27T01:19:55.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:55.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:55 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:55.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-75"}]: dispatch 2024-04-27T01:19:55.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:55 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-75"}]: dispatch 2024-04-27T01:19:55.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:55 smithi162 ceph-mon[37680]: osdmap e433: 8 total, 8 up, 8 in 2024-04-27T01:19:55.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:55.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:55 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:55.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-75"}]: dispatch 2024-04-27T01:19:55.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:55 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-75"}]: dispatch 2024-04-27T01:19:55.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:55 smithi162 ceph-mon[26714]: osdmap e433: 8 total, 8 up, 8 in 2024-04-27T01:19:55.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:55.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:55 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-75"}]: dispatch 2024-04-27T01:19:55.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:55 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-75"}]: dispatch 2024-04-27T01:19:56.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:56 smithi181 ceph-mon[30237]: pgmap v645: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:19:56.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:56 smithi181 ceph-mon[30237]: osdmap e434: 8 total, 8 up, 8 in 2024-04-27T01:19:56.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:56 smithi162 ceph-mon[37680]: pgmap v645: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:19:56.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:56 smithi162 ceph-mon[37680]: osdmap e434: 8 total, 8 up, 8 in 2024-04-27T01:19:56.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:56 smithi162 ceph-mon[26714]: pgmap v645: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:19:56.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:56 smithi162 ceph-mon[26714]: osdmap e434: 8 total, 8 up, 8 in 2024-04-27T01:19:57.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:57 smithi181 ceph-mon[30237]: osdmap e435: 8 total, 8 up, 8 in 2024-04-27T01:19:57.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:57.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:57 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:57.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:57 smithi162 ceph-mon[37680]: osdmap e435: 8 total, 8 up, 8 in 2024-04-27T01:19:57.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:57.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:57 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:57.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:57 smithi162 ceph-mon[26714]: osdmap e435: 8 total, 8 up, 8 in 2024-04-27T01:19:57.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:57.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:57 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:19:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:58 smithi181 ceph-mon[30237]: pgmap v648: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:19:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:58 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:58 smithi181 ceph-mon[30237]: osdmap e436: 8 total, 8 up, 8 in 2024-04-27T01:19:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:58.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:58 smithi162 ceph-mon[37680]: pgmap v648: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:19:58.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:58 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:58.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:58.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:58 smithi162 ceph-mon[37680]: osdmap e436: 8 total, 8 up, 8 in 2024-04-27T01:19:58.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:58.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:58 smithi162 ceph-mon[26714]: pgmap v648: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:19:58.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:58 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:19:58.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:19:58.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:58 smithi162 ceph-mon[26714]: osdmap e436: 8 total, 8 up, 8 in 2024-04-27T01:19:58.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:19:59.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-mon[26714]: osdmap e437: 8 total, 8 up, 8 in 2024-04-27T01:19:59.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:59.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:59.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-77"}]: dispatch 2024-04-27T01:19:59.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-77"}]: dispatch 2024-04-27T01:19:59.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-mon[37680]: osdmap e437: 8 total, 8 up, 8 in 2024-04-27T01:19:59.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:59.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:59.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-77"}]: dispatch 2024-04-27T01:19:59.729 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-77"}]: dispatch 2024-04-27T01:19:59.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:59 smithi181 ceph-mon[30237]: osdmap e437: 8 total, 8 up, 8 in 2024-04-27T01:19:59.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:59.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:59 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:19:59.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-77"}]: dispatch 2024-04-27T01:19:59.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:19:59 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-77"}]: dispatch 2024-04-27T01:20:00.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:19:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:19:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:20:00.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:00 smithi181 ceph-mon[30237]: pgmap v651: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:20:00.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:00 smithi181 ceph-mon[30237]: osdmap e438: 8 total, 8 up, 8 in 2024-04-27T01:20:00.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:00 smithi181 ceph-mon[30237]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-27T01:20:00.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:00 smithi181 ceph-mon[30237]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-27T01:20:00.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:00 smithi181 ceph-mon[30237]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-27T01:20:00.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:00 smithi181 ceph-mon[30237]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-27T01:20:00.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[37680]: pgmap v651: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:20:00.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[37680]: osdmap e438: 8 total, 8 up, 8 in 2024-04-27T01:20:00.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[37680]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-27T01:20:00.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[37680]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-27T01:20:00.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[37680]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-27T01:20:00.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[37680]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-27T01:20:00.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[26714]: pgmap v651: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:20:00.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[26714]: osdmap e438: 8 total, 8 up, 8 in 2024-04-27T01:20:00.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[26714]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-27T01:20:00.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[26714]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-27T01:20:00.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[26714]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-27T01:20:00.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:00 smithi162 ceph-mon[26714]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-27T01:20:01.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:01 smithi181 ceph-mon[30237]: osdmap e439: 8 total, 8 up, 8 in 2024-04-27T01:20:01.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:01.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:01 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:01.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:01 smithi162 ceph-mon[37680]: osdmap e439: 8 total, 8 up, 8 in 2024-04-27T01:20:01.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:01.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:01 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:01.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:01 smithi162 ceph-mon[26714]: osdmap e439: 8 total, 8 up, 8 in 2024-04-27T01:20:01.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:01.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:01 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:02.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:20:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:20:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:02 smithi181 ceph-mon[30237]: pgmap v654: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:02 smithi181 ceph-mon[30237]: osdmap e440: 8 total, 8 up, 8 in 2024-04-27T01:20:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:02.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:02 smithi162 ceph-mon[37680]: pgmap v654: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:02.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:02.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:02 smithi162 ceph-mon[37680]: osdmap e440: 8 total, 8 up, 8 in 2024-04-27T01:20:02.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:02.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:02 smithi162 ceph-mon[26714]: pgmap v654: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:02.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:02.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:02 smithi162 ceph-mon[26714]: osdmap e440: 8 total, 8 up, 8 in 2024-04-27T01:20:02.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:03.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:03 smithi181 ceph-mon[30237]: osdmap e441: 8 total, 8 up, 8 in 2024-04-27T01:20:03.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:03.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:03 smithi162 ceph-mon[37680]: osdmap e441: 8 total, 8 up, 8 in 2024-04-27T01:20:03.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:03.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:03 smithi162 ceph-mon[26714]: osdmap e441: 8 total, 8 up, 8 in 2024-04-27T01:20:03.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:04 smithi181 ceph-mon[30237]: pgmap v657: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-04-27T01:20:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:04 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:20:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:04 smithi181 ceph-mon[30237]: osdmap e442: 8 total, 8 up, 8 in 2024-04-27T01:20:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-04-27T01:20:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:04 smithi181 ceph-mon[30237]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-04-27T01:20:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:04 smithi181 ceph-mon[30237]: 305.8 deep-scrub starts 2024-04-27T01:20:04.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:04 smithi181 ceph-mon[30237]: 305.8 deep-scrub ok 2024-04-27T01:20:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[37680]: pgmap v657: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-04-27T01:20:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:20:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[37680]: osdmap e442: 8 total, 8 up, 8 in 2024-04-27T01:20:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-04-27T01:20:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[37680]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-04-27T01:20:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[37680]: 305.8 deep-scrub starts 2024-04-27T01:20:04.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[37680]: 305.8 deep-scrub ok 2024-04-27T01:20:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[26714]: pgmap v657: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 1.2 KiB/s wr, 181 op/s 2024-04-27T01:20:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:20:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[26714]: osdmap e442: 8 total, 8 up, 8 in 2024-04-27T01:20:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-04-27T01:20:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[26714]: from='mon.? v1:172.21.15.181:0/3958199638' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.8"}]: dispatch 2024-04-27T01:20:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[26714]: 305.8 deep-scrub starts 2024-04-27T01:20:04.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:04 smithi162 ceph-mon[26714]: 305.8 deep-scrub ok 2024-04-27T01:20:05.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:05 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:05.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:05 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:05.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:05 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:06.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:06 smithi181 ceph-mon[30237]: pgmap v659: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 1.0 KiB/s wr, 152 op/s 2024-04-27T01:20:06.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:06 smithi162 ceph-mon[37680]: pgmap v659: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 1.0 KiB/s wr, 152 op/s 2024-04-27T01:20:06.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:06 smithi162 ceph-mon[26714]: pgmap v659: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 1.0 KiB/s wr, 152 op/s 2024-04-27T01:20:08.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:08 smithi181 ceph-mon[30237]: pgmap v660: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 1.2 KiB/s wr, 123 op/s 2024-04-27T01:20:08.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:08 smithi162 ceph-mon[37680]: pgmap v660: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 1.2 KiB/s wr, 123 op/s 2024-04-27T01:20:08.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:08 smithi162 ceph-mon[26714]: pgmap v660: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 1.2 KiB/s wr, 123 op/s 2024-04-27T01:20:10.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:20:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:20:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:20:10.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:10 smithi181 ceph-mon[30237]: pgmap v661: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 921 B/s wr, 95 op/s 2024-04-27T01:20:10.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:10 smithi162 ceph-mon[37680]: pgmap v661: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 921 B/s wr, 95 op/s 2024-04-27T01:20:10.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:10 smithi162 ceph-mon[26714]: pgmap v661: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 921 B/s wr, 95 op/s 2024-04-27T01:20:12.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:12 smithi181 ceph-mon[30237]: pgmap v662: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 233 B/s wr, 1 op/s 2024-04-27T01:20:12.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:20:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:20:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:12 smithi162 ceph-mon[37680]: pgmap v662: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 233 B/s wr, 1 op/s 2024-04-27T01:20:12.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:12 smithi162 ceph-mon[26714]: pgmap v662: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 233 B/s wr, 1 op/s 2024-04-27T01:20:13.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:13.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:14.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:14.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:14 smithi162 ceph-mon[37680]: pgmap v663: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-04-27T01:20:14.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:14 smithi162 ceph-mon[26714]: pgmap v663: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-04-27T01:20:15.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:14 smithi181 ceph-mon[30237]: pgmap v663: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-04-27T01:20:15.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:15 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:15.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:15 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:16.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:15 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:20:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:16 smithi162 ceph-mon[37680]: pgmap v664: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 173 B/s wr, 1 op/s 2024-04-27T01:20:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:20:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:20:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:20:16.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:16 smithi162 ceph-mon[26714]: pgmap v664: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 173 B/s wr, 1 op/s 2024-04-27T01:20:16.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:20:16.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:20:16.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:20:17.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:16 smithi181 ceph-mon[30237]: pgmap v664: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 173 B/s wr, 1 op/s 2024-04-27T01:20:17.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:20:17.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:20:17.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:20:18.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:18 smithi162 ceph-mon[37680]: pgmap v665: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-27T01:20:18.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:18 smithi162 ceph-mon[26714]: pgmap v665: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-27T01:20:19.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:18 smithi181 ceph-mon[30237]: pgmap v665: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-27T01:20:20.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:20:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:20:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:20:20.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:20 smithi162 ceph-mon[37680]: pgmap v666: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:20:20.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:20.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:20 smithi162 ceph-mon[26714]: pgmap v666: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:20:20.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:21.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:20 smithi181 ceph-mon[30237]: pgmap v666: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:20:21.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:22.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:22 smithi162 ceph-mon[37680]: pgmap v667: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:20:22.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:22 smithi162 ceph-mon[26714]: pgmap v667: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:20:23.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:22 smithi181 ceph-mon[30237]: pgmap v667: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:20:23.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:20:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:20:23.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:23.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:23.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:23 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:23.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-79"}]: dispatch 2024-04-27T01:20:23.836 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:20:23 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0[41669]: 2024-04-27T01:20:23.537+0000 7fda5137d700 -1 snap_mapper.add_oid found existing snaps mapped on 305:140d259d:test-rados-api-smithi162-74551-80::foo:2, removing 2024-04-27T01:20:23.837 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:20:23 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1[47531]: 2024-04-27T01:20:23.537+0000 7fb9b6a7d700 -1 snap_mapper.add_oid found existing snaps mapped on 305:140d259d:test-rados-api-smithi162-74551-80::foo:2, removing 2024-04-27T01:20:23.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:23.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:23.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:23 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:23.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-79"}]: dispatch 2024-04-27T01:20:24.008 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:20:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:20:23.537+0000 7f05bbd37700 -1 snap_mapper.add_oid found existing snaps mapped on 305:140d259d:test-rados-api-smithi162-74551-80::foo:2, removing 2024-04-27T01:20:24.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:24.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:24.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:23 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:24.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-79"}]: dispatch 2024-04-27T01:20:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:24 smithi162 ceph-mon[37680]: pgmap v668: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:20:24.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:24 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-79"}]: dispatch 2024-04-27T01:20:24.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:24 smithi162 ceph-mon[26714]: pgmap v668: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:20:24.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:24 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-79"}]: dispatch 2024-04-27T01:20:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:24 smithi181 ceph-mon[30237]: pgmap v668: 228 pgs: 228 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:20:25.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:24 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-79"}]: dispatch 2024-04-27T01:20:25.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:25 smithi162 ceph-mon[37680]: osdmap e443: 8 total, 8 up, 8 in 2024-04-27T01:20:25.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:25 smithi162 ceph-mon[26714]: osdmap e443: 8 total, 8 up, 8 in 2024-04-27T01:20:26.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:25 smithi181 ceph-mon[30237]: osdmap e443: 8 total, 8 up, 8 in 2024-04-27T01:20:26.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:26 smithi162 ceph-mon[37680]: pgmap v670: 196 pgs: 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:20:26.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:26 smithi162 ceph-mon[37680]: osdmap e444: 8 total, 8 up, 8 in 2024-04-27T01:20:26.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:26.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:26 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:26.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:26 smithi162 ceph-mon[26714]: pgmap v670: 196 pgs: 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:20:26.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:26 smithi162 ceph-mon[26714]: osdmap e444: 8 total, 8 up, 8 in 2024-04-27T01:20:26.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:26.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:26 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:27.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:26 smithi181 ceph-mon[30237]: pgmap v670: 196 pgs: 196 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:20:27.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:26 smithi181 ceph-mon[30237]: osdmap e444: 8 total, 8 up, 8 in 2024-04-27T01:20:27.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:27.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:26 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:27.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:27 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:27.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:27 smithi162 ceph-mon[37680]: osdmap e445: 8 total, 8 up, 8 in 2024-04-27T01:20:27.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:27.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:27 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:27.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:27 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:27.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:27 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:27.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:27 smithi162 ceph-mon[26714]: osdmap e445: 8 total, 8 up, 8 in 2024-04-27T01:20:27.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:27.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:27 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:27.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:27 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:28.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:27 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:28.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:27 smithi181 ceph-mon[30237]: osdmap e445: 8 total, 8 up, 8 in 2024-04-27T01:20:28.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:28.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:27 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:28.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:27 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:28.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:28 smithi162 ceph-mon[26714]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-27T01:20:28.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:28.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:28.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:28 smithi162 ceph-mon[26714]: osdmap e446: 8 total, 8 up, 8 in 2024-04-27T01:20:28.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:28 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:28.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:28 smithi162 ceph-mon[37680]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-27T01:20:28.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:28.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:28.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:28 smithi162 ceph-mon[37680]: osdmap e446: 8 total, 8 up, 8 in 2024-04-27T01:20:28.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:28 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:29.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:28 smithi181 ceph-mon[30237]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-27T01:20:29.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:29.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:29.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:28 smithi181 ceph-mon[30237]: osdmap e446: 8 total, 8 up, 8 in 2024-04-27T01:20:29.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:28 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:29.835 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:20:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:20:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:20:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[37680]: osdmap e447: 8 total, 8 up, 8 in 2024-04-27T01:20:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[37680]: osdmap e448: 8 total, 8 up, 8 in 2024-04-27T01:20:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:29.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:20:29.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[26714]: osdmap e447: 8 total, 8 up, 8 in 2024-04-27T01:20:29.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:29.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:29.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:29.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[26714]: osdmap e448: 8 total, 8 up, 8 in 2024-04-27T01:20:29.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:30.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:29 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:20:30.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:29 smithi181 ceph-mon[30237]: osdmap e447: 8 total, 8 up, 8 in 2024-04-27T01:20:30.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:30.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:29 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:30.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:29 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:30.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:29 smithi181 ceph-mon[30237]: osdmap e448: 8 total, 8 up, 8 in 2024-04-27T01:20:30.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:30.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:30 smithi162 ceph-mon[37680]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:20:30.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:30 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:30.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:30 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:30.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:30 smithi162 ceph-mon[37680]: osdmap e449: 8 total, 8 up, 8 in 2024-04-27T01:20:30.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:30 smithi162 ceph-mon[26714]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:20:30.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:30 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:30.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:30 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:30.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:30 smithi162 ceph-mon[26714]: osdmap e449: 8 total, 8 up, 8 in 2024-04-27T01:20:31.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:30 smithi181 ceph-mon[30237]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:20:31.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:30 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:31.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:30 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:31.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:30 smithi181 ceph-mon[30237]: osdmap e449: 8 total, 8 up, 8 in 2024-04-27T01:20:32.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-27T01:20:32.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:31 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-27T01:20:32.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-04-27T01:20:32.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:31 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-04-27T01:20:32.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-27T01:20:32.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:31 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-27T01:20:32.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-04-27T01:20:32.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:31 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-04-27T01:20:32.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-27T01:20:32.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:31 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-27T01:20:32.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-04-27T01:20:32.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:31 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]: dispatch 2024-04-27T01:20:32.909 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:20:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:20:32.910 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:32 smithi181 ceph-mon[30237]: pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:32.910 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:32 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-27T01:20:32.910 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:32 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]': finished 2024-04-27T01:20:32.910 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:32 smithi181 ceph-mon[30237]: osdmap e450: 8 total, 8 up, 8 in 2024-04-27T01:20:32.910 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-27T01:20:32.910 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:32 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-27T01:20:33.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[37680]: pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:33.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-27T01:20:33.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]': finished 2024-04-27T01:20:33.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[37680]: osdmap e450: 8 total, 8 up, 8 in 2024-04-27T01:20:33.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-27T01:20:33.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-27T01:20:33.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[26714]: pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:33.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-27T01:20:33.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "306.3", "id": [7, 2]}]': finished 2024-04-27T01:20:33.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[26714]: osdmap e450: 8 total, 8 up, 8 in 2024-04-27T01:20:33.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-27T01:20:33.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:32 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-27T01:20:34.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:34.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:33 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-27T01:20:34.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:33 smithi181 ceph-mon[30237]: osdmap e451: 8 total, 8 up, 8 in 2024-04-27T01:20:34.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:34.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:33 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:34.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:34.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:33 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-27T01:20:34.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:33 smithi162 ceph-mon[37680]: osdmap e451: 8 total, 8 up, 8 in 2024-04-27T01:20:34.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:34.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:33 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:34.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:34.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:33 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-27T01:20:34.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:33 smithi162 ceph-mon[26714]: osdmap e451: 8 total, 8 up, 8 in 2024-04-27T01:20:34.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:34.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:33 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:35.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:34 smithi181 ceph-mon[30237]: pgmap v682: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-04-27T01:20:35.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:35.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:34 smithi181 ceph-mon[30237]: osdmap e452: 8 total, 8 up, 8 in 2024-04-27T01:20:35.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:35.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:35.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-81"}]: dispatch 2024-04-27T01:20:35.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:34 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-81"}]: dispatch 2024-04-27T01:20:35.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[37680]: pgmap v682: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-04-27T01:20:35.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:35.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[37680]: osdmap e452: 8 total, 8 up, 8 in 2024-04-27T01:20:35.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:35.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:35.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-81"}]: dispatch 2024-04-27T01:20:35.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-81"}]: dispatch 2024-04-27T01:20:35.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[26714]: pgmap v682: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 9.2 KiB/s wr, 21 op/s 2024-04-27T01:20:35.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:35.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[26714]: osdmap e452: 8 total, 8 up, 8 in 2024-04-27T01:20:35.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:35.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:35.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-81"}]: dispatch 2024-04-27T01:20:35.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:34 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-81"}]: dispatch 2024-04-27T01:20:36.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:35 smithi181 ceph-mon[30237]: osdmap e453: 8 total, 8 up, 8 in 2024-04-27T01:20:36.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:35 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:36.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:35 smithi162 ceph-mon[37680]: osdmap e453: 8 total, 8 up, 8 in 2024-04-27T01:20:36.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:35 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:36.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:35 smithi162 ceph-mon[26714]: osdmap e453: 8 total, 8 up, 8 in 2024-04-27T01:20:36.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:35 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:37.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:36 smithi181 ceph-mon[30237]: pgmap v685: 196 pgs: 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 7.0 KiB/s wr, 21 op/s 2024-04-27T01:20:37.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:36 smithi181 ceph-mon[30237]: osdmap e454: 8 total, 8 up, 8 in 2024-04-27T01:20:37.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:37.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:36 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:37.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:36 smithi162 ceph-mon[37680]: pgmap v685: 196 pgs: 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 7.0 KiB/s wr, 21 op/s 2024-04-27T01:20:37.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:36 smithi162 ceph-mon[37680]: osdmap e454: 8 total, 8 up, 8 in 2024-04-27T01:20:37.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:37.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:36 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:37.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:36 smithi162 ceph-mon[26714]: pgmap v685: 196 pgs: 196 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 7.0 KiB/s wr, 21 op/s 2024-04-27T01:20:37.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:36 smithi162 ceph-mon[26714]: osdmap e454: 8 total, 8 up, 8 in 2024-04-27T01:20:37.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:37.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:36 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:38.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:37 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:38.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:37 smithi181 ceph-mon[30237]: osdmap e455: 8 total, 8 up, 8 in 2024-04-27T01:20:38.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:38.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:38.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:37 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:38.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:37 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:38.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:37 smithi162 ceph-mon[37680]: osdmap e455: 8 total, 8 up, 8 in 2024-04-27T01:20:38.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:38.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:38.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:37 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:38.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:37 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:38.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:37 smithi162 ceph-mon[26714]: osdmap e455: 8 total, 8 up, 8 in 2024-04-27T01:20:38.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:38.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:38.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:37 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:39.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:38 smithi181 ceph-mon[30237]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 KiB/s wr, 18 op/s 2024-04-27T01:20:39.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:38 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:39.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:39.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:38 smithi181 ceph-mon[30237]: osdmap e456: 8 total, 8 up, 8 in 2024-04-27T01:20:39.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:38 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:39.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:38 smithi162 ceph-mon[37680]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 KiB/s wr, 18 op/s 2024-04-27T01:20:39.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:38 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:39.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:39.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:38 smithi162 ceph-mon[37680]: osdmap e456: 8 total, 8 up, 8 in 2024-04-27T01:20:39.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:38 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:39.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:38 smithi162 ceph-mon[26714]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 KiB/s wr, 18 op/s 2024-04-27T01:20:39.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:38 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:39.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:39.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:38 smithi162 ceph-mon[26714]: osdmap e456: 8 total, 8 up, 8 in 2024-04-27T01:20:39.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:38 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:20:40.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:40.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:39 smithi181 ceph-mon[30237]: osdmap e457: 8 total, 8 up, 8 in 2024-04-27T01:20:40.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:40.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:40.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:39 smithi181 ceph-mon[30237]: osdmap e458: 8 total, 8 up, 8 in 2024-04-27T01:20:40.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:40.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:40.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:20:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:20:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:20:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[26714]: osdmap e457: 8 total, 8 up, 8 in 2024-04-27T01:20:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[26714]: osdmap e458: 8 total, 8 up, 8 in 2024-04-27T01:20:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:40.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:20:40.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:40.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[37680]: osdmap e457: 8 total, 8 up, 8 in 2024-04-27T01:20:40.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:40.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:40.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[37680]: osdmap e458: 8 total, 8 up, 8 in 2024-04-27T01:20:40.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:40.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:41.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:40 smithi181 ceph-mon[30237]: pgmap v691: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-04-27T01:20:41.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:40 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:41.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:40 smithi181 ceph-mon[30237]: osdmap e459: 8 total, 8 up, 8 in 2024-04-27T01:20:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:40 smithi162 ceph-mon[37680]: pgmap v691: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-04-27T01:20:41.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:40 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:41.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:40 smithi162 ceph-mon[37680]: osdmap e459: 8 total, 8 up, 8 in 2024-04-27T01:20:41.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:40 smithi162 ceph-mon[26714]: pgmap v691: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.2 KiB/s wr, 19 op/s 2024-04-27T01:20:41.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:40 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:41.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:40 smithi162 ceph-mon[26714]: osdmap e459: 8 total, 8 up, 8 in 2024-04-27T01:20:43.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:42 smithi181 ceph-mon[30237]: pgmap v694: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:43.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:42 smithi181 ceph-mon[30237]: osdmap e460: 8 total, 8 up, 8 in 2024-04-27T01:20:43.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:20:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:20:43.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:42 smithi162 ceph-mon[26714]: pgmap v694: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:43.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:42 smithi162 ceph-mon[26714]: osdmap e460: 8 total, 8 up, 8 in 2024-04-27T01:20:43.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:42 smithi162 ceph-mon[37680]: pgmap v694: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:43.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:42 smithi162 ceph-mon[37680]: osdmap e460: 8 total, 8 up, 8 in 2024-04-27T01:20:44.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:44.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:43 smithi181 ceph-mon[30237]: osdmap e461: 8 total, 8 up, 8 in 2024-04-27T01:20:44.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:44.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:43 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:44.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-83"}]: dispatch 2024-04-27T01:20:44.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:43 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-83"}]: dispatch 2024-04-27T01:20:44.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[37680]: osdmap e461: 8 total, 8 up, 8 in 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-83"}]: dispatch 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-83"}]: dispatch 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[26714]: osdmap e461: 8 total, 8 up, 8 in 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-83"}]: dispatch 2024-04-27T01:20:44.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:43 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-83"}]: dispatch 2024-04-27T01:20:45.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:44 smithi181 ceph-mon[30237]: pgmap v697: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-27T01:20:45.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:44 smithi181 ceph-mon[30237]: osdmap e462: 8 total, 8 up, 8 in 2024-04-27T01:20:45.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:44 smithi162 ceph-mon[37680]: pgmap v697: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-27T01:20:45.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:44 smithi162 ceph-mon[37680]: osdmap e462: 8 total, 8 up, 8 in 2024-04-27T01:20:45.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:44 smithi162 ceph-mon[26714]: pgmap v697: 228 pgs: 228 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-27T01:20:45.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:44 smithi162 ceph-mon[26714]: osdmap e462: 8 total, 8 up, 8 in 2024-04-27T01:20:46.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:45 smithi181 ceph-mon[30237]: osdmap e463: 8 total, 8 up, 8 in 2024-04-27T01:20:46.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:46.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:45 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:46.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:45 smithi162 ceph-mon[37680]: osdmap e463: 8 total, 8 up, 8 in 2024-04-27T01:20:46.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:46.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:45 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:46.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:45 smithi162 ceph-mon[26714]: osdmap e463: 8 total, 8 up, 8 in 2024-04-27T01:20:46.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:46.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:45 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:47.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:46 smithi181 ceph-mon[30237]: pgmap v700: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:20:47.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:46 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:20:47.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:46 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:47.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:46 smithi181 ceph-mon[30237]: osdmap e464: 8 total, 8 up, 8 in 2024-04-27T01:20:47.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:47.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:47.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:46 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:47.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[37680]: pgmap v700: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:20:47.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:20:47.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:47.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[37680]: osdmap e464: 8 total, 8 up, 8 in 2024-04-27T01:20:47.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:47.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:47.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:47.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[26714]: pgmap v700: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:20:47.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:20:47.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:47.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[26714]: osdmap e464: 8 total, 8 up, 8 in 2024-04-27T01:20:47.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:47.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:47.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:46 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:48.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:47 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:48.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:48.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:47 smithi181 ceph-mon[30237]: osdmap e465: 8 total, 8 up, 8 in 2024-04-27T01:20:48.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:47 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:48.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:47 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:48.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:48.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:47 smithi162 ceph-mon[26714]: osdmap e465: 8 total, 8 up, 8 in 2024-04-27T01:20:48.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:47 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:47 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:47 smithi162 ceph-mon[37680]: osdmap e465: 8 total, 8 up, 8 in 2024-04-27T01:20:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:47 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:49.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:48 smithi181 ceph-mon[30237]: pgmap v703: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-27T01:20:49.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:48 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:20:49.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:49.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:48 smithi181 ceph-mon[30237]: osdmap e466: 8 total, 8 up, 8 in 2024-04-27T01:20:49.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:48 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:49.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:48 smithi162 ceph-mon[26714]: pgmap v703: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-27T01:20:49.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:48 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:20:49.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:49.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:48 smithi162 ceph-mon[26714]: osdmap e466: 8 total, 8 up, 8 in 2024-04-27T01:20:49.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:48 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:49.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:48 smithi162 ceph-mon[37680]: pgmap v703: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-04-27T01:20:49.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:48 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_tier","val": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:20:49.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:49.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:48 smithi162 ceph-mon[37680]: osdmap e466: 8 total, 8 up, 8 in 2024-04-27T01:20:49.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:48 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:49 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:50.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:50.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:49 smithi181 ceph-mon[30237]: osdmap e467: 8 total, 8 up, 8 in 2024-04-27T01:20:50.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:49 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:50.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:20:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:20:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:20:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:49 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:49 smithi162 ceph-mon[26714]: osdmap e467: 8 total, 8 up, 8 in 2024-04-27T01:20:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:49 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:50.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:49 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:50.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:50.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:49 smithi162 ceph-mon[37680]: osdmap e467: 8 total, 8 up, 8 in 2024-04-27T01:20:50.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:49 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:50 smithi181 ceph-mon[30237]: pgmap v706: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-27T01:20:51.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:50 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:51.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:50 smithi181 ceph-mon[30237]: osdmap e468: 8 total, 8 up, 8 in 2024-04-27T01:20:51.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:51.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:50 smithi162 ceph-mon[26714]: pgmap v706: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-27T01:20:51.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:50 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:51.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:50 smithi162 ceph-mon[26714]: osdmap e468: 8 total, 8 up, 8 in 2024-04-27T01:20:51.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:51.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:50 smithi162 ceph-mon[37680]: pgmap v706: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-27T01:20:51.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:50 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:20:51.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:50 smithi162 ceph-mon[37680]: osdmap e468: 8 total, 8 up, 8 in 2024-04-27T01:20:51.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:20:52.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:51 smithi181 ceph-mon[30237]: osdmap e469: 8 total, 8 up, 8 in 2024-04-27T01:20:52.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:51 smithi162 ceph-mon[26714]: osdmap e469: 8 total, 8 up, 8 in 2024-04-27T01:20:52.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:51 smithi162 ceph-mon[37680]: osdmap e469: 8 total, 8 up, 8 in 2024-04-27T01:20:53.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:20:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:20:53.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:52 smithi181 ceph-mon[30237]: pgmap v709: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:53.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:52 smithi181 ceph-mon[30237]: osdmap e470: 8 total, 8 up, 8 in 2024-04-27T01:20:53.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:53.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:52 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:53.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-85"}]: dispatch 2024-04-27T01:20:53.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:52 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-85"}]: dispatch 2024-04-27T01:20:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[37680]: pgmap v709: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[37680]: osdmap e470: 8 total, 8 up, 8 in 2024-04-27T01:20:53.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:53.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:53.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-85"}]: dispatch 2024-04-27T01:20:53.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-85"}]: dispatch 2024-04-27T01:20:53.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[26714]: pgmap v709: 228 pgs: 228 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:20:53.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[26714]: osdmap e470: 8 total, 8 up, 8 in 2024-04-27T01:20:53.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:53.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:20:53.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-85"}]: dispatch 2024-04-27T01:20:53.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:52 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-85"}]: dispatch 2024-04-27T01:20:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:53 smithi162 ceph-mon[37680]: osdmap e471: 8 total, 8 up, 8 in 2024-04-27T01:20:54.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:54.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:53 smithi162 ceph-mon[26714]: osdmap e471: 8 total, 8 up, 8 in 2024-04-27T01:20:54.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:20:54.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:53 smithi181 ceph-mon[30237]: osdmap e471: 8 total, 8 up, 8 in 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18061 ms) 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22637 ms) 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3037 ms) 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3016 ms) 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3012 ms) 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-04-27T01:20:54.782 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3010 ms) 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4029 ms) 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38163 ms) 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16785 ms) 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4023 ms) 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-04-27T01:20:54.783 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3023 ms) 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24134 ms) 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15055 ms) 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5049 ms) 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4026 ms) 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.784 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: waiting for scrubs... 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: done waiting 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25075 ms) 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10081 ms) 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9078 ms) 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9060 ms) 2024-04-27T01:20:54.785 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-04-27T01:20:55.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:54 smithi162 ceph-mon[37680]: pgmap v712: 196 pgs: 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-04-27T01:20:55.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:54 smithi162 ceph-mon[37680]: osdmap e472: 8 total, 8 up, 8 in 2024-04-27T01:20:55.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:55.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:54 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:55.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:54 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:20:55.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:54 smithi162 ceph-mon[26714]: pgmap v712: 196 pgs: 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-04-27T01:20:55.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:54 smithi162 ceph-mon[26714]: osdmap e472: 8 total, 8 up, 8 in 2024-04-27T01:20:55.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:55.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:54 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:55.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:54 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:20:55.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:54 smithi181 ceph-mon[30237]: pgmap v712: 196 pgs: 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-04-27T01:20:55.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:54 smithi181 ceph-mon[30237]: osdmap e472: 8 total, 8 up, 8 in 2024-04-27T01:20:55.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:55.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:54 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:20:55.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:54 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:20:56.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:55 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:56.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:55 smithi162 ceph-mon[26714]: osdmap e473: 8 total, 8 up, 8 in 2024-04-27T01:20:56.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:56.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:55 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:56.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:55 smithi162 ceph-mon[37680]: osdmap e473: 8 total, 8 up, 8 in 2024-04-27T01:20:56.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:56.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:55 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:20:56.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:55 smithi181 ceph-mon[30237]: osdmap e473: 8 total, 8 up, 8 in 2024-04-27T01:20:56.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:20:57.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:56 smithi162 ceph-mon[26714]: pgmap v715: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-04-27T01:20:57.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:56 smithi162 ceph-mon[26714]: osdmap e474: 8 total, 8 up, 8 in 2024-04-27T01:20:57.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:57.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:57.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:56 smithi162 ceph-mon[37680]: pgmap v715: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-04-27T01:20:57.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:56 smithi162 ceph-mon[37680]: osdmap e474: 8 total, 8 up, 8 in 2024-04-27T01:20:57.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:57.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:57.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:56 smithi181 ceph-mon[30237]: pgmap v715: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-04-27T01:20:57.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:56 smithi181 ceph-mon[30237]: osdmap e474: 8 total, 8 up, 8 in 2024-04-27T01:20:57.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:57.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:20:58.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:58.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[37680]: osdmap e475: 8 total, 8 up, 8 in 2024-04-27T01:20:58.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_tier","val": "test-rados-api-smithi162-74551-89-test-flush"}]: dispatch 2024-04-27T01:20:58.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_tier","val": "test-rados-api-smithi162-74551-89-test-flush"}]: dispatch 2024-04-27T01:20:58.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_tier","val": "test-rados-api-smithi162-74551-89-test-flush"}]': finished 2024-04-27T01:20:58.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[37680]: osdmap e476: 8 total, 8 up, 8 in 2024-04-27T01:20:58.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:58.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:58.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:58.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[26714]: osdmap e475: 8 total, 8 up, 8 in 2024-04-27T01:20:58.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_tier","val": "test-rados-api-smithi162-74551-89-test-flush"}]: dispatch 2024-04-27T01:20:58.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_tier","val": "test-rados-api-smithi162-74551-89-test-flush"}]: dispatch 2024-04-27T01:20:58.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_tier","val": "test-rados-api-smithi162-74551-89-test-flush"}]': finished 2024-04-27T01:20:58.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[26714]: osdmap e476: 8 total, 8 up, 8 in 2024-04-27T01:20:58.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:58.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:57 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:58.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:57 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:20:58.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:57 smithi181 ceph-mon[30237]: osdmap e475: 8 total, 8 up, 8 in 2024-04-27T01:20:58.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_tier","val": "test-rados-api-smithi162-74551-89-test-flush"}]: dispatch 2024-04-27T01:20:58.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:57 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_tier","val": "test-rados-api-smithi162-74551-89-test-flush"}]: dispatch 2024-04-27T01:20:58.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:57 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_tier","val": "test-rados-api-smithi162-74551-89-test-flush"}]': finished 2024-04-27T01:20:58.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:57 smithi181 ceph-mon[30237]: osdmap e476: 8 total, 8 up, 8 in 2024-04-27T01:20:58.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:58.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:57 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:20:59.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:58 smithi162 ceph-mon[37680]: pgmap v718: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:20:59.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:59.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:59.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:58 smithi162 ceph-mon[37680]: osdmap e477: 8 total, 8 up, 8 in 2024-04-27T01:20:59.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:59.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:58 smithi162 ceph-mon[26714]: pgmap v718: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:20:59.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:59.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:59.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:58 smithi162 ceph-mon[26714]: osdmap e477: 8 total, 8 up, 8 in 2024-04-27T01:20:59.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:59.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:58 smithi181 ceph-mon[30237]: pgmap v718: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:20:59.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:20:59.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:20:59.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:58 smithi181 ceph-mon[30237]: osdmap e477: 8 total, 8 up, 8 in 2024-04-27T01:20:59.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:21:00.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:20:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:20:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:21:00.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:59 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:21:00.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:59 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:21:00.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:20:59 smithi162 ceph-mon[37680]: osdmap e478: 8 total, 8 up, 8 in 2024-04-27T01:21:00.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:59 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:21:00.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:59 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:21:00.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:20:59 smithi162 ceph-mon[26714]: osdmap e478: 8 total, 8 up, 8 in 2024-04-27T01:21:00.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:59 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:21:00.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:59 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:21:00.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:20:59 smithi181 ceph-mon[30237]: osdmap e478: 8 total, 8 up, 8 in 2024-04-27T01:21:01.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:00 smithi162 ceph-mon[37680]: pgmap v721: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:21:01.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:00 smithi162 ceph-mon[26714]: pgmap v721: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:21:01.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:00 smithi181 ceph-mon[30237]: pgmap v721: 260 pgs: 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:21:02.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:01 smithi162 ceph-mon[37680]: osdmap e479: 8 total, 8 up, 8 in 2024-04-27T01:21:02.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:02.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:01 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:02.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-87"}]: dispatch 2024-04-27T01:21:02.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:01 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-87"}]: dispatch 2024-04-27T01:21:02.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:01 smithi162 ceph-mon[26714]: osdmap e479: 8 total, 8 up, 8 in 2024-04-27T01:21:02.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:02.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:01 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:02.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-87"}]: dispatch 2024-04-27T01:21:02.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:01 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-87"}]: dispatch 2024-04-27T01:21:02.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:01 smithi181 ceph-mon[30237]: osdmap e479: 8 total, 8 up, 8 in 2024-04-27T01:21:02.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:02.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:01 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:02.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-87"}]: dispatch 2024-04-27T01:21:02.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:01 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-87"}]: dispatch 2024-04-27T01:21:03.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:21:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:21:03.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:02 smithi181 ceph-mon[30237]: pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:03.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:02 smithi181 ceph-mon[30237]: osdmap e480: 8 total, 8 up, 8 in 2024-04-27T01:21:03.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:02 smithi162 ceph-mon[37680]: pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:03.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:02 smithi162 ceph-mon[37680]: osdmap e480: 8 total, 8 up, 8 in 2024-04-27T01:21:03.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:02 smithi162 ceph-mon[26714]: pgmap v724: 228 pgs: 228 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:03.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:02 smithi162 ceph-mon[26714]: osdmap e480: 8 total, 8 up, 8 in 2024-04-27T01:21:04.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:04.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:03 smithi181 ceph-mon[30237]: osdmap e481: 8 total, 8 up, 8 in 2024-04-27T01:21:04.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:04.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:04.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:04.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:03 smithi181 ceph-mon[30237]: osdmap e482: 8 total, 8 up, 8 in 2024-04-27T01:21:04.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:04.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[37680]: osdmap e481: 8 total, 8 up, 8 in 2024-04-27T01:21:04.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:04.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:04.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:04.338 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[37680]: osdmap e482: 8 total, 8 up, 8 in 2024-04-27T01:21:04.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:04.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[26714]: osdmap e481: 8 total, 8 up, 8 in 2024-04-27T01:21:04.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:04.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:04.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:04.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:03 smithi162 ceph-mon[26714]: osdmap e482: 8 total, 8 up, 8 in 2024-04-27T01:21:05.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:04 smithi181 ceph-mon[30237]: pgmap v727: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:21:05.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:05.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:21:05.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:21:05.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:21:05.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:04 smithi181 ceph-mon[30237]: osdmap e483: 8 total, 8 up, 8 in 2024-04-27T01:21:05.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[37680]: pgmap v727: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:21:05.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:05.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:21:05.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:21:05.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:21:05.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[37680]: osdmap e483: 8 total, 8 up, 8 in 2024-04-27T01:21:05.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[26714]: pgmap v727: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:21:05.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:05.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:21:05.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:21:05.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:21:05.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:04 smithi162 ceph-mon[26714]: osdmap e483: 8 total, 8 up, 8 in 2024-04-27T01:21:06.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:05 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:06.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:05 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:06.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:05 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:07.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:06 smithi181 ceph-mon[30237]: pgmap v730: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:21:07.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:06 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:21:07.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:06 smithi181 ceph-mon[30237]: osdmap e484: 8 total, 8 up, 8 in 2024-04-27T01:21:07.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:06 smithi162 ceph-mon[26714]: pgmap v730: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:21:07.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:06 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:21:07.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:06 smithi162 ceph-mon[26714]: osdmap e484: 8 total, 8 up, 8 in 2024-04-27T01:21:07.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:06 smithi162 ceph-mon[37680]: pgmap v730: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:21:07.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:06 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:21:07.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:06 smithi162 ceph-mon[37680]: osdmap e484: 8 total, 8 up, 8 in 2024-04-27T01:21:08.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:07 smithi181 ceph-mon[30237]: osdmap e485: 8 total, 8 up, 8 in 2024-04-27T01:21:08.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:08.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:07 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:08.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-90"}]: dispatch 2024-04-27T01:21:08.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:07 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-90"}]: dispatch 2024-04-27T01:21:08.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:07 smithi162 ceph-mon[26714]: osdmap e485: 8 total, 8 up, 8 in 2024-04-27T01:21:08.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:08.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:07 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:08.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-90"}]: dispatch 2024-04-27T01:21:08.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:07 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-90"}]: dispatch 2024-04-27T01:21:08.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:07 smithi162 ceph-mon[37680]: osdmap e485: 8 total, 8 up, 8 in 2024-04-27T01:21:08.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:08.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:07 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:08.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-90"}]: dispatch 2024-04-27T01:21:08.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:07 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-90"}]: dispatch 2024-04-27T01:21:09.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:08 smithi181 ceph-mon[30237]: pgmap v733: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 9 op/s 2024-04-27T01:21:09.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:08 smithi181 ceph-mon[30237]: osdmap e486: 8 total, 8 up, 8 in 2024-04-27T01:21:09.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:08 smithi162 ceph-mon[37680]: pgmap v733: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 9 op/s 2024-04-27T01:21:09.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:08 smithi162 ceph-mon[37680]: osdmap e486: 8 total, 8 up, 8 in 2024-04-27T01:21:09.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:08 smithi162 ceph-mon[26714]: pgmap v733: 228 pgs: 228 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 9 op/s 2024-04-27T01:21:09.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:08 smithi162 ceph-mon[26714]: osdmap e486: 8 total, 8 up, 8 in 2024-04-27T01:21:10.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:21:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:21:10.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-mon[37680]: osdmap e487: 8 total, 8 up, 8 in 2024-04-27T01:21:10.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:10.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:10.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:10.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-mon[37680]: osdmap e488: 8 total, 8 up, 8 in 2024-04-27T01:21:10.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-mon[26714]: osdmap e487: 8 total, 8 up, 8 in 2024-04-27T01:21:10.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:10.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:10.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:10.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:09 smithi162 ceph-mon[26714]: osdmap e488: 8 total, 8 up, 8 in 2024-04-27T01:21:10.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:09 smithi181 ceph-mon[30237]: osdmap e487: 8 total, 8 up, 8 in 2024-04-27T01:21:10.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:10.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:10.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:10.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:09 smithi181 ceph-mon[30237]: osdmap e488: 8 total, 8 up, 8 in 2024-04-27T01:21:11.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:10 smithi181 ceph-mon[30237]: pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-27T01:21:11.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:11.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:10 smithi181 ceph-mon[30237]: osdmap e489: 8 total, 8 up, 8 in 2024-04-27T01:21:11.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:10 smithi162 ceph-mon[37680]: pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-27T01:21:11.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:11.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:10 smithi162 ceph-mon[37680]: osdmap e489: 8 total, 8 up, 8 in 2024-04-27T01:21:11.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:10 smithi162 ceph-mon[26714]: pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-27T01:21:11.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:11.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:10 smithi162 ceph-mon[26714]: osdmap e489: 8 total, 8 up, 8 in 2024-04-27T01:21:12.886 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:21:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:21:13.137 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[37680]: pgmap v739: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:13.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[37680]: osdmap e490: 8 total, 8 up, 8 in 2024-04-27T01:21:13.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:21:13.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:13.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:13.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-92"}]: dispatch 2024-04-27T01:21:13.138 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-92"}]: dispatch 2024-04-27T01:21:13.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[26714]: pgmap v739: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:13.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[26714]: osdmap e490: 8 total, 8 up, 8 in 2024-04-27T01:21:13.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:21:13.138 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:13.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:13.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-92"}]: dispatch 2024-04-27T01:21:13.139 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-92"}]: dispatch 2024-04-27T01:21:13.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:12 smithi181 ceph-mon[30237]: pgmap v739: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:13.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:12 smithi181 ceph-mon[30237]: osdmap e490: 8 total, 8 up, 8 in 2024-04-27T01:21:13.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:12 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:21:13.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:13.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:13.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-92"}]: dispatch 2024-04-27T01:21:13.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-92"}]: dispatch 2024-04-27T01:21:14.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:14.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:13 smithi181 ceph-mon[30237]: osdmap e491: 8 total, 8 up, 8 in 2024-04-27T01:21:14.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:14.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:13 smithi162 ceph-mon[37680]: osdmap e491: 8 total, 8 up, 8 in 2024-04-27T01:21:14.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:14.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:13 smithi162 ceph-mon[26714]: osdmap e491: 8 total, 8 up, 8 in 2024-04-27T01:21:15.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:14 smithi181 ceph-mon[30237]: pgmap v742: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:21:15.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:14 smithi181 ceph-mon[30237]: osdmap e492: 8 total, 8 up, 8 in 2024-04-27T01:21:15.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:15.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:14 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:15.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:14 smithi162 ceph-mon[26714]: pgmap v742: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:21:15.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:14 smithi162 ceph-mon[26714]: osdmap e492: 8 total, 8 up, 8 in 2024-04-27T01:21:15.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:15.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:14 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:15.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:14 smithi162 ceph-mon[37680]: pgmap v742: 196 pgs: 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:21:15.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:14 smithi162 ceph-mon[37680]: osdmap e492: 8 total, 8 up, 8 in 2024-04-27T01:21:15.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:15.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:14 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:16.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:15 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:16.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:15 smithi181 ceph-mon[30237]: osdmap e493: 8 total, 8 up, 8 in 2024-04-27T01:21:16.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:16.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:15 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:21:16.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:15 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:16.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:15 smithi162 ceph-mon[37680]: osdmap e493: 8 total, 8 up, 8 in 2024-04-27T01:21:16.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:16.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:15 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:21:16.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:15 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:16.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:15 smithi162 ceph-mon[26714]: osdmap e493: 8 total, 8 up, 8 in 2024-04-27T01:21:16.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:16.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:15 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:21:17.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: pgmap v745: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: osdmap e494: 8 total, 8 up, 8 in 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:21:17.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:21:17.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:21:17.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:21:17.259 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:16 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: pgmap v745: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: osdmap e494: 8 total, 8 up, 8 in 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:21:17.337 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: pgmap v745: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: osdmap e494: 8 total, 8 up, 8 in 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.338 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-27T01:21:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:21:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:21:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:21:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:21:17.339 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:16 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:21:18.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:17 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi181 to 3248M 2024-04-27T01:21:18.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:17 smithi181 ceph-mon[30237]: Adjusting osd_memory_target on smithi162 to 3248M 2024-04-27T01:21:18.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:17 smithi181 ceph-mon[30237]: osdmap e495: 8 total, 8 up, 8 in 2024-04-27T01:21:18.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:17 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi181 to 3248M 2024-04-27T01:21:18.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:17 smithi162 ceph-mon[37680]: Adjusting osd_memory_target on smithi162 to 3248M 2024-04-27T01:21:18.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:17 smithi162 ceph-mon[37680]: osdmap e495: 8 total, 8 up, 8 in 2024-04-27T01:21:18.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:17 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi181 to 3248M 2024-04-27T01:21:18.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:17 smithi162 ceph-mon[26714]: Adjusting osd_memory_target on smithi162 to 3248M 2024-04-27T01:21:18.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:17 smithi162 ceph-mon[26714]: osdmap e495: 8 total, 8 up, 8 in 2024-04-27T01:21:19.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:18 smithi181 ceph-mon[30237]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-27T01:21:19.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:18 smithi162 ceph-mon[37680]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-27T01:21:19.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:18 smithi162 ceph-mon[26714]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-04-27T01:21:20.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:21:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:21:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:21:21.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:20 smithi181 ceph-mon[30237]: pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-27T01:21:21.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:21.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:20 smithi162 ceph-mon[37680]: pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-27T01:21:21.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:21.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:20 smithi162 ceph-mon[26714]: pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-04-27T01:21:21.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:22.953 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:21:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:21:23.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:22 smithi181 ceph-mon[30237]: pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-27T01:21:23.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:22 smithi162 ceph-mon[37680]: pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-27T01:21:23.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:22 smithi162 ceph-mon[26714]: pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-04-27T01:21:24.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:24.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:24.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:25.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:24 smithi181 ceph-mon[30237]: pgmap v751: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:21:25.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:24 smithi162 ceph-mon[37680]: pgmap v751: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:21:25.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:24 smithi162 ceph-mon[26714]: pgmap v751: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:21:27.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:26 smithi181 ceph-mon[30237]: pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 875 B/s wr, 3 op/s 2024-04-27T01:21:27.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:26 smithi162 ceph-mon[37680]: pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 875 B/s wr, 3 op/s 2024-04-27T01:21:27.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:26 smithi162 ceph-mon[26714]: pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 875 B/s wr, 3 op/s 2024-04-27T01:21:29.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:28 smithi181 ceph-mon[30237]: pgmap v753: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 791 B/s wr, 3 op/s 2024-04-27T01:21:29.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:28 smithi181 ceph-mon[30237]: osdmap e496: 8 total, 8 up, 8 in 2024-04-27T01:21:29.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:28 smithi162 ceph-mon[37680]: pgmap v753: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 791 B/s wr, 3 op/s 2024-04-27T01:21:29.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:28 smithi162 ceph-mon[37680]: osdmap e496: 8 total, 8 up, 8 in 2024-04-27T01:21:29.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:28 smithi162 ceph-mon[26714]: pgmap v753: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 791 B/s wr, 3 op/s 2024-04-27T01:21:29.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:28 smithi162 ceph-mon[26714]: osdmap e496: 8 total, 8 up, 8 in 2024-04-27T01:21:29.757 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:21:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=cleanup t=2024-04-27T01:21:29.286415177Z level=info msg="Completed cleanup jobs" duration=2.258378ms 2024-04-27T01:21:29.757 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:21:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=sqlstore.transactions t=2024-04-27T01:21:29.295021129Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-27T01:21:30.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:21:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:21:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:21:31.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:30 smithi181 ceph-mon[30237]: pgmap v755: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:21:31.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:30 smithi162 ceph-mon[26714]: pgmap v755: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:21:31.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:30 smithi162 ceph-mon[37680]: pgmap v755: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:21:32.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:31 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.14", "id": [0, 1]}]: dispatch 2024-04-27T01:21:32.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:31 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.14", "id": [0, 1]}]: dispatch 2024-04-27T01:21:32.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:31 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.14", "id": [0, 1]}]: dispatch 2024-04-27T01:21:32.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:31 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.14", "id": [0, 1]}]: dispatch 2024-04-27T01:21:32.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:31 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.14", "id": [0, 1]}]: dispatch 2024-04-27T01:21:32.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:31 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.14", "id": [0, 1]}]: dispatch 2024-04-27T01:21:32.994 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:21:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:21:33.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:33 smithi181 ceph-mon[30237]: pgmap v756: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:21:33.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:33 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.14", "id": [0, 1]}]': finished 2024-04-27T01:21:33.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:33 smithi181 ceph-mon[30237]: osdmap e497: 8 total, 8 up, 8 in 2024-04-27T01:21:33.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:33 smithi162 ceph-mon[37680]: pgmap v756: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:21:33.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:33 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.14", "id": [0, 1]}]': finished 2024-04-27T01:21:33.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:33 smithi162 ceph-mon[37680]: osdmap e497: 8 total, 8 up, 8 in 2024-04-27T01:21:33.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:33 smithi162 ceph-mon[26714]: pgmap v756: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:21:33.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:33 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.14", "id": [0, 1]}]': finished 2024-04-27T01:21:33.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:33 smithi162 ceph-mon[26714]: osdmap e497: 8 total, 8 up, 8 in 2024-04-27T01:21:34.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:34 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:34.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:34 smithi181 ceph-mon[30237]: osdmap e498: 8 total, 8 up, 8 in 2024-04-27T01:21:34.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:34 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:34.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:34 smithi162 ceph-mon[37680]: osdmap e498: 8 total, 8 up, 8 in 2024-04-27T01:21:34.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:34 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:34.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:34 smithi162 ceph-mon[26714]: osdmap e498: 8 total, 8 up, 8 in 2024-04-27T01:21:35.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:35 smithi181 ceph-mon[30237]: pgmap v759: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:21:35.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:35 smithi181 ceph-mon[30237]: osdmap e499: 8 total, 8 up, 8 in 2024-04-27T01:21:35.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:35 smithi162 ceph-mon[37680]: pgmap v759: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:21:35.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:35 smithi162 ceph-mon[37680]: osdmap e499: 8 total, 8 up, 8 in 2024-04-27T01:21:35.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:35 smithi162 ceph-mon[26714]: pgmap v759: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:21:35.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:35 smithi162 ceph-mon[26714]: osdmap e499: 8 total, 8 up, 8 in 2024-04-27T01:21:36.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:36.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:36.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:37.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:37 smithi162 ceph-mon[37680]: pgmap v761: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:21:37.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:37 smithi162 ceph-mon[26714]: pgmap v761: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:21:37.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:37 smithi181 ceph-mon[30237]: pgmap v761: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:21:39.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-mon[37680]: pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:21:39.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:39.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:39.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-94"}]: dispatch 2024-04-27T01:21:39.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-94"}]: dispatch 2024-04-27T01:21:39.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-mon[26714]: pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:21:39.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:39.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:39.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-94"}]: dispatch 2024-04-27T01:21:39.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-94"}]: dispatch 2024-04-27T01:21:39.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:39 smithi181 ceph-mon[30237]: pgmap v762: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:21:39.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:39.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:39.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-94"}]: dispatch 2024-04-27T01:21:39.508 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:39 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-94"}]: dispatch 2024-04-27T01:21:40.025 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:21:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:21:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:21:40.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:40 smithi162 ceph-mon[26714]: osdmap e500: 8 total, 8 up, 8 in 2024-04-27T01:21:40.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:40 smithi162 ceph-mon[37680]: osdmap e500: 8 total, 8 up, 8 in 2024-04-27T01:21:40.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:40 smithi181 ceph-mon[30237]: osdmap e500: 8 total, 8 up, 8 in 2024-04-27T01:21:41.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:41 smithi162 ceph-mon[37680]: pgmap v764: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:41.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:41 smithi162 ceph-mon[37680]: osdmap e501: 8 total, 8 up, 8 in 2024-04-27T01:21:41.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:41.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:41 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:41.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:41 smithi162 ceph-mon[26714]: pgmap v764: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:41.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:41 smithi162 ceph-mon[26714]: osdmap e501: 8 total, 8 up, 8 in 2024-04-27T01:21:41.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:41.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:41 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:41.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:41 smithi181 ceph-mon[30237]: pgmap v764: 196 pgs: 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:41.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:41 smithi181 ceph-mon[30237]: osdmap e501: 8 total, 8 up, 8 in 2024-04-27T01:21:41.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:41.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:41 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:42.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:42 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:42.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:42 smithi162 ceph-mon[26714]: osdmap e502: 8 total, 8 up, 8 in 2024-04-27T01:21:42.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:42.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:42 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:42.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:42 smithi162 ceph-mon[37680]: osdmap e502: 8 total, 8 up, 8 in 2024-04-27T01:21:42.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:42.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:42 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:42.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:42 smithi181 ceph-mon[30237]: osdmap e502: 8 total, 8 up, 8 in 2024-04-27T01:21:42.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:21:43.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:21:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:21:43.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:43 smithi162 ceph-mon[37680]: pgmap v767: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:43.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:43 smithi162 ceph-mon[37680]: osdmap e503: 8 total, 8 up, 8 in 2024-04-27T01:21:43.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:43 smithi162 ceph-mon[26714]: pgmap v767: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:43.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:43 smithi162 ceph-mon[26714]: osdmap e503: 8 total, 8 up, 8 in 2024-04-27T01:21:43.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:43 smithi181 ceph-mon[30237]: pgmap v767: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:21:43.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:43 smithi181 ceph-mon[30237]: osdmap e503: 8 total, 8 up, 8 in 2024-04-27T01:21:44.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:44 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:44.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:44 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:44.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:44 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:45.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:45 smithi162 ceph-mon[26714]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-27T01:21:45.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:45 smithi162 ceph-mon[37680]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-27T01:21:45.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:45 smithi181 ceph-mon[30237]: pgmap v769: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-27T01:21:47.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:47 smithi162 ceph-mon[26714]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 2 op/s 2024-04-27T01:21:47.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:47 smithi162 ceph-mon[37680]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 2 op/s 2024-04-27T01:21:47.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:47 smithi181 ceph-mon[30237]: pgmap v770: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 2 op/s 2024-04-27T01:21:48.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:48 smithi162 ceph-mon[26714]: pgmap v771: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 845 B/s wr, 3 op/s 2024-04-27T01:21:48.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:48 smithi162 ceph-mon[37680]: pgmap v771: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 845 B/s wr, 3 op/s 2024-04-27T01:21:48.507 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:48 smithi181 ceph-mon[30237]: pgmap v771: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 845 B/s wr, 3 op/s 2024-04-27T01:21:50.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:21:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:21:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:21:50.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:50 smithi181 ceph-mon[30237]: pgmap v772: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 743 B/s wr, 2 op/s 2024-04-27T01:21:50.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:50.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:50 smithi162 ceph-mon[37680]: pgmap v772: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 743 B/s wr, 2 op/s 2024-04-27T01:21:50.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:50.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:50 smithi162 ceph-mon[26714]: pgmap v772: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 743 B/s wr, 2 op/s 2024-04-27T01:21:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:21:52.335 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0[41669]: 2024-04-27T01:21:52.096+0000 7fda4eb78700 -1 snap_mapper.add_oid found existing snaps mapped on 108:b52e754d:test-rados-api-smithi162-74551-97::foo:21, removing 2024-04-27T01:21:52.336 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1[47531]: 2024-04-27T01:21:52.096+0000 7fb9b4278700 -1 snap_mapper.add_oid found existing snaps mapped on 108:b52e754d:test-rados-api-smithi162-74551-97::foo:21, removing 2024-04-27T01:21:52.336 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3[59259]: 2024-04-27T01:21:52.096+0000 7f0d04c5b700 -1 snap_mapper.add_oid found existing snaps mapped on 108:b52e754d:test-rados-api-smithi162-74551-97::foo:21, removing 2024-04-27T01:21:52.660 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:52 smithi181 ceph-mon[30237]: pgmap v773: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 614 B/s wr, 2 op/s 2024-04-27T01:21:52.660 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:52.660 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:52 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:52.661 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-96"}]: dispatch 2024-04-27T01:21:52.661 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:52 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-96"}]: dispatch 2024-04-27T01:21:52.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-mon[26714]: pgmap v773: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 614 B/s wr, 2 op/s 2024-04-27T01:21:52.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:52.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:52.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-96"}]: dispatch 2024-04-27T01:21:52.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-96"}]: dispatch 2024-04-27T01:21:52.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-mon[37680]: pgmap v773: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 614 B/s wr, 2 op/s 2024-04-27T01:21:52.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:52.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:21:52.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-96"}]: dispatch 2024-04-27T01:21:52.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:52 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-96"}]: dispatch 2024-04-27T01:21:53.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:21:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:21:53.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:53 smithi181 ceph-mon[30237]: osdmap e504: 8 total, 8 up, 8 in 2024-04-27T01:21:53.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:53.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:53 smithi162 ceph-mon[37680]: osdmap e504: 8 total, 8 up, 8 in 2024-04-27T01:21:53.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:53.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:53 smithi162 ceph-mon[26714]: osdmap e504: 8 total, 8 up, 8 in 2024-04-27T01:21:53.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:21:54.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:54 smithi181 ceph-mon[30237]: pgmap v775: 196 pgs: 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-04-27T01:21:54.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:54 smithi181 ceph-mon[30237]: osdmap e505: 8 total, 8 up, 8 in 2024-04-27T01:21:54.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:54.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:54 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:54.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:54 smithi162 ceph-mon[37680]: pgmap v775: 196 pgs: 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-04-27T01:21:54.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:54 smithi162 ceph-mon[37680]: osdmap e505: 8 total, 8 up, 8 in 2024-04-27T01:21:54.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:54.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:54 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:54.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:54 smithi162 ceph-mon[26714]: pgmap v775: 196 pgs: 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-04-27T01:21:54.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:54 smithi162 ceph-mon[26714]: osdmap e505: 8 total, 8 up, 8 in 2024-04-27T01:21:54.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:54.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:54 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:21:55.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:55 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:55.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:55 smithi181 ceph-mon[30237]: osdmap e506: 8 total, 8 up, 8 in 2024-04-27T01:21:55.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:21:55.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:55 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:21:55.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:55 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:55.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:55 smithi162 ceph-mon[37680]: osdmap e506: 8 total, 8 up, 8 in 2024-04-27T01:21:55.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:21:55.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:55 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:21:55.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:55 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:21:55.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:55 smithi162 ceph-mon[26714]: osdmap e506: 8 total, 8 up, 8 in 2024-04-27T01:21:55.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:21:55.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:55 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:21:56.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:56 smithi181 ceph-mon[30237]: pgmap v778: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:21:56.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:21:56.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:56 smithi181 ceph-mon[30237]: osdmap e507: 8 total, 8 up, 8 in 2024-04-27T01:21:56.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-98", "mode": "writeback"}]: dispatch 2024-04-27T01:21:56.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:56 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-98", "mode": "writeback"}]: dispatch 2024-04-27T01:21:56.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:56 smithi162 ceph-mon[26714]: pgmap v778: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:21:56.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:21:56.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:56 smithi162 ceph-mon[26714]: osdmap e507: 8 total, 8 up, 8 in 2024-04-27T01:21:56.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-98", "mode": "writeback"}]: dispatch 2024-04-27T01:21:56.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:56 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-98", "mode": "writeback"}]: dispatch 2024-04-27T01:21:56.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:56 smithi162 ceph-mon[37680]: pgmap v778: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:21:56.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:21:56.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:56 smithi162 ceph-mon[37680]: osdmap e507: 8 total, 8 up, 8 in 2024-04-27T01:21:56.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-98", "mode": "writeback"}]: dispatch 2024-04-27T01:21:56.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:56 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-98", "mode": "writeback"}]: dispatch 2024-04-27T01:21:57.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:57 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:21:57.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:57 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-98", "mode": "writeback"}]': finished 2024-04-27T01:21:57.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:57 smithi181 ceph-mon[30237]: osdmap e508: 8 total, 8 up, 8 in 2024-04-27T01:21:57.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:21:57.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:57 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:21:57.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:57 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-98"}]': finished 2024-04-27T01:21:57.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:57 smithi181 ceph-mon[30237]: osdmap e509: 8 total, 8 up, 8 in 2024-04-27T01:21:57.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:21:57.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-98", "mode": "writeback"}]': finished 2024-04-27T01:21:57.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[37680]: osdmap e508: 8 total, 8 up, 8 in 2024-04-27T01:21:57.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:21:57.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:21:57.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-98"}]': finished 2024-04-27T01:21:57.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[37680]: osdmap e509: 8 total, 8 up, 8 in 2024-04-27T01:21:57.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:21:57.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-98", "mode": "writeback"}]': finished 2024-04-27T01:21:57.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[26714]: osdmap e508: 8 total, 8 up, 8 in 2024-04-27T01:21:57.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:21:57.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:21:57.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-98"}]': finished 2024-04-27T01:21:57.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:57 smithi162 ceph-mon[26714]: osdmap e509: 8 total, 8 up, 8 in 2024-04-27T01:21:58.405 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[37680]: pgmap v781: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[26714]: pgmap v781: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[26714]: osdmap e510: 8 total, 8 up, 8 in 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-27T01:21:58.406 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-27T01:21:58.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:58 smithi181 ceph-mon[30237]: pgmap v781: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:21:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:21:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:21:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:58 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:21:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:21:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:58 smithi181 ceph-mon[30237]: osdmap e510: 8 total, 8 up, 8 in 2024-04-27T01:21:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-27T01:21:58.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:21:58 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-27T01:21:58.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:21:58.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[37680]: osdmap e510: 8 total, 8 up, 8 in 2024-04-27T01:21:58.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-27T01:21:58.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:21:58 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-27T01:22:00.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:21:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:21:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:22:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:00 smithi181 ceph-mon[30237]: pgmap v784: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:22:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:00 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_count","val": "1"}]': finished 2024-04-27T01:22:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:00 smithi181 ceph-mon[30237]: osdmap e511: 8 total, 8 up, 8 in 2024-04-27T01:22:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:00.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:00 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:00.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:00 smithi162 ceph-mon[37680]: pgmap v784: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:22:00.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:00 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_count","val": "1"}]': finished 2024-04-27T01:22:00.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:00 smithi162 ceph-mon[37680]: osdmap e511: 8 total, 8 up, 8 in 2024-04-27T01:22:00.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:00.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:00 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:00.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:00 smithi162 ceph-mon[26714]: pgmap v784: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:22:00.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:00 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_count","val": "1"}]': finished 2024-04-27T01:22:00.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:00 smithi162 ceph-mon[26714]: osdmap e511: 8 total, 8 up, 8 in 2024-04-27T01:22:00.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:00.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:00 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:01.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:01 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:22:01.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-27T01:22:01.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:01 smithi181 ceph-mon[30237]: osdmap e512: 8 total, 8 up, 8 in 2024-04-27T01:22:01.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:01 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-27T01:22:01.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:01 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "target_max_objects","val": "250"}]': finished 2024-04-27T01:22:01.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:01 smithi181 ceph-mon[30237]: osdmap e513: 8 total, 8 up, 8 in 2024-04-27T01:22:01.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:22:01.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-27T01:22:01.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[37680]: osdmap e512: 8 total, 8 up, 8 in 2024-04-27T01:22:01.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-27T01:22:01.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "target_max_objects","val": "250"}]': finished 2024-04-27T01:22:01.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[37680]: osdmap e513: 8 total, 8 up, 8 in 2024-04-27T01:22:01.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:22:01.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-27T01:22:01.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[26714]: osdmap e512: 8 total, 8 up, 8 in 2024-04-27T01:22:01.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-27T01:22:01.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-98","var": "target_max_objects","val": "250"}]': finished 2024-04-27T01:22:01.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:01 smithi162 ceph-mon[26714]: osdmap e513: 8 total, 8 up, 8 in 2024-04-27T01:22:02.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:22:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:22:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:02 smithi181 ceph-mon[30237]: pgmap v787: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:02.759 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:02 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:02 smithi162 ceph-mon[26714]: pgmap v787: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:02 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:02.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:02 smithi162 ceph-mon[37680]: pgmap v787: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:02.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:02.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:02 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:22:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:22:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:03 smithi181 ceph-mon[30237]: osdmap e514: 8 total, 8 up, 8 in 2024-04-27T01:22:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:03 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:22:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:03.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:22:03.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:22:03.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:03 smithi162 ceph-mon[37680]: osdmap e514: 8 total, 8 up, 8 in 2024-04-27T01:22:03.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:03 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:22:03.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:03.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:22:03.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:22:03.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:03 smithi162 ceph-mon[26714]: osdmap e514: 8 total, 8 up, 8 in 2024-04-27T01:22:03.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:03 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98"}]: dispatch 2024-04-27T01:22:03.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:05.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:04 smithi181 ceph-mon[30237]: pgmap v790: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:05.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:04 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98"}]': finished 2024-04-27T01:22:05.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:04 smithi181 ceph-mon[30237]: osdmap e515: 8 total, 8 up, 8 in 2024-04-27T01:22:05.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:04 smithi162 ceph-mon[37680]: pgmap v790: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:05.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:04 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98"}]': finished 2024-04-27T01:22:05.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:04 smithi162 ceph-mon[37680]: osdmap e515: 8 total, 8 up, 8 in 2024-04-27T01:22:05.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:04 smithi162 ceph-mon[26714]: pgmap v790: 228 pgs: 228 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:05.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:04 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-98"}]': finished 2024-04-27T01:22:05.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:04 smithi162 ceph-mon[26714]: osdmap e515: 8 total, 8 up, 8 in 2024-04-27T01:22:06.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:05 smithi181 ceph-mon[30237]: osdmap e516: 8 total, 8 up, 8 in 2024-04-27T01:22:06.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:05 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:06.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:05 smithi181 ceph-mon[30237]: osdmap e517: 8 total, 8 up, 8 in 2024-04-27T01:22:06.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:06.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:05 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:06.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:05 smithi162 ceph-mon[37680]: osdmap e516: 8 total, 8 up, 8 in 2024-04-27T01:22:06.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:05 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:06.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:05 smithi162 ceph-mon[37680]: osdmap e517: 8 total, 8 up, 8 in 2024-04-27T01:22:06.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:06.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:05 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:06.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:05 smithi162 ceph-mon[26714]: osdmap e516: 8 total, 8 up, 8 in 2024-04-27T01:22:06.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:05 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:06.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:05 smithi162 ceph-mon[26714]: osdmap e517: 8 total, 8 up, 8 in 2024-04-27T01:22:06.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:06.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:05 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:07.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:06 smithi181 ceph-mon[30237]: pgmap v793: 196 pgs: 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:07.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:06 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:07.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:06 smithi181 ceph-mon[30237]: osdmap e518: 8 total, 8 up, 8 in 2024-04-27T01:22:07.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:22:07.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:06 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:22:07.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:06 smithi162 ceph-mon[37680]: pgmap v793: 196 pgs: 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:07.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:06 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:07.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:06 smithi162 ceph-mon[37680]: osdmap e518: 8 total, 8 up, 8 in 2024-04-27T01:22:07.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:22:07.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:06 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:22:07.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:06 smithi162 ceph-mon[26714]: pgmap v793: 196 pgs: 196 active+clean; 8.4 MiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:07.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:06 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:07.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:06 smithi162 ceph-mon[26714]: osdmap e518: 8 total, 8 up, 8 in 2024-04-27T01:22:07.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:22:07.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:06 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:22:08.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:08 smithi181 ceph-mon[30237]: pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:08.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:22:08.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:08 smithi181 ceph-mon[30237]: osdmap e519: 8 total, 8 up, 8 in 2024-04-27T01:22:08.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:08.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:08 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:08.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:08 smithi162 ceph-mon[37680]: pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:08.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:22:08.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:08 smithi162 ceph-mon[37680]: osdmap e519: 8 total, 8 up, 8 in 2024-04-27T01:22:08.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:08.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:08 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:08.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:08 smithi162 ceph-mon[26714]: pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:08.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:22:08.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:08 smithi162 ceph-mon[26714]: osdmap e519: 8 total, 8 up, 8 in 2024-04-27T01:22:08.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:08.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:08 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:09.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-100"}]': finished 2024-04-27T01:22:09.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[37680]: osdmap e520: 8 total, 8 up, 8 in 2024-04-27T01:22:09.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-100", "mode": "writeback"}]: dispatch 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-100", "mode": "writeback"}]: dispatch 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-100", "mode": "writeback"}]': finished 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[37680]: osdmap e521: 8 total, 8 up, 8 in 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-100"}]': finished 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[26714]: osdmap e520: 8 total, 8 up, 8 in 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-100", "mode": "writeback"}]: dispatch 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-100", "mode": "writeback"}]: dispatch 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-100", "mode": "writeback"}]': finished 2024-04-27T01:22:09.728 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-mon[26714]: osdmap e521: 8 total, 8 up, 8 in 2024-04-27T01:22:09.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-6", "overlaypool": "test-rados-api-smithi162-74551-100"}]': finished 2024-04-27T01:22:09.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:09 smithi181 ceph-mon[30237]: osdmap e520: 8 total, 8 up, 8 in 2024-04-27T01:22:09.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-100", "mode": "writeback"}]: dispatch 2024-04-27T01:22:09.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-100", "mode": "writeback"}]: dispatch 2024-04-27T01:22:09.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:09 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:22:09.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:09 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-100", "mode": "writeback"}]': finished 2024-04-27T01:22:09.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:09 smithi181 ceph-mon[30237]: osdmap e521: 8 total, 8 up, 8 in 2024-04-27T01:22:10.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:22:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:22:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:22:10.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:10 smithi181 ceph-mon[30237]: pgmap v799: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:10.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:10.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:10.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:22:10.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:10 smithi181 ceph-mon[30237]: osdmap e522: 8 total, 8 up, 8 in 2024-04-27T01:22:10.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:10.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:10 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:10.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[26714]: pgmap v799: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:10.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:10.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:10.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:22:10.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[26714]: osdmap e522: 8 total, 8 up, 8 in 2024-04-27T01:22:10.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:10.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:10.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[37680]: pgmap v799: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:10.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:10.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:10.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:22:10.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[37680]: osdmap e522: 8 total, 8 up, 8 in 2024-04-27T01:22:10.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:10.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:10 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:12.757 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:22:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:22:12.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:12 smithi181 ceph-mon[30237]: pgmap v802: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:12.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:22:12.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:12 smithi181 ceph-mon[30237]: osdmap e523: 8 total, 8 up, 8 in 2024-04-27T01:22:12.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:12.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:12 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:12 smithi162 ceph-mon[37680]: pgmap v802: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:22:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:12 smithi162 ceph-mon[37680]: osdmap e523: 8 total, 8 up, 8 in 2024-04-27T01:22:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:12 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:12.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:12 smithi162 ceph-mon[26714]: pgmap v802: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:12.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:22:12.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:12 smithi162 ceph-mon[26714]: osdmap e523: 8 total, 8 up, 8 in 2024-04-27T01:22:12.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:12.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:12 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:22:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:22:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[37680]: osdmap e524: 8 total, 8 up, 8 in 2024-04-27T01:22:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-27T01:22:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-27T01:22:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-27T01:22:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[37680]: osdmap e525: 8 total, 8 up, 8 in 2024-04-27T01:22:13.738 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:22:13.738 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:22:13.738 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[26714]: osdmap e524: 8 total, 8 up, 8 in 2024-04-27T01:22:13.738 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-27T01:22:13.738 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-27T01:22:13.738 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:13.738 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-27T01:22:13.738 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:13 smithi162 ceph-mon[26714]: osdmap e525: 8 total, 8 up, 8 in 2024-04-27T01:22:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:13 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:22:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:13 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:22:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:13 smithi181 ceph-mon[30237]: osdmap e524: 8 total, 8 up, 8 in 2024-04-27T01:22:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-27T01:22:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:13 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-27T01:22:13.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:13.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:13 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-27T01:22:13.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:13 smithi181 ceph-mon[30237]: osdmap e525: 8 total, 8 up, 8 in 2024-04-27T01:22:14.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:14 smithi181 ceph-mon[30237]: pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:14.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:14.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:14 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:14.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:14 smithi162 ceph-mon[37680]: pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:14.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:14.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:14 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:14.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:14 smithi162 ceph-mon[26714]: pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:14.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:14.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:14 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:15.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:15 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:22:15.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:15 smithi181 ceph-mon[30237]: osdmap e526: 8 total, 8 up, 8 in 2024-04-27T01:22:15.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:15.758 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:15 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:15.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:15 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:22:15.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:15 smithi162 ceph-mon[37680]: osdmap e526: 8 total, 8 up, 8 in 2024-04-27T01:22:15.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:15.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:15 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:15.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:15 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]': finished 2024-04-27T01:22:15.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:15 smithi162 ceph-mon[26714]: osdmap e526: 8 total, 8 up, 8 in 2024-04-27T01:22:15.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:15.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:15 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100"}]: dispatch 2024-04-27T01:22:16.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:16 smithi181 ceph-mon[30237]: pgmap v808: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:16.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:16 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100"}]': finished 2024-04-27T01:22:16.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:16 smithi181 ceph-mon[30237]: osdmap e527: 8 total, 8 up, 8 in 2024-04-27T01:22:16.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:16 smithi162 ceph-mon[37680]: pgmap v808: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:16.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:16 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100"}]': finished 2024-04-27T01:22:16.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:16 smithi162 ceph-mon[37680]: osdmap e527: 8 total, 8 up, 8 in 2024-04-27T01:22:16.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:16 smithi162 ceph-mon[26714]: pgmap v808: 228 pgs: 228 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:16.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:16 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-100"}]': finished 2024-04-27T01:22:16.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:16 smithi162 ceph-mon[26714]: osdmap e527: 8 total, 8 up, 8 in 2024-04-27T01:22:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:17 smithi181 ceph-mon[30237]: osdmap e528: 8 total, 8 up, 8 in 2024-04-27T01:22:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:17 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:22:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:17 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:22:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:17 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:22:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:17 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:22:17.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:17 smithi162 ceph-mon[37680]: osdmap e528: 8 total, 8 up, 8 in 2024-04-27T01:22:17.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:17 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:22:17.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:17 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:22:17.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:17 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:22:17.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:17 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:22:17.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:17 smithi162 ceph-mon[26714]: osdmap e528: 8 total, 8 up, 8 in 2024-04-27T01:22:17.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:17 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:22:17.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:17 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:22:17.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:17 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:22:17.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:17 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:22:18.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:18 smithi162 ceph-mon[37680]: pgmap v811: 196 pgs: 196 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:18.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:18 smithi162 ceph-mon[37680]: osdmap e529: 8 total, 8 up, 8 in 2024-04-27T01:22:18.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:18.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:18 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:18.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:18 smithi162 ceph-mon[26714]: pgmap v811: 196 pgs: 196 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:18.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:18 smithi162 ceph-mon[26714]: osdmap e529: 8 total, 8 up, 8 in 2024-04-27T01:22:18.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:18.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:18 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:19.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:18 smithi181 ceph-mon[30237]: pgmap v811: 196 pgs: 196 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:19.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:18 smithi181 ceph-mon[30237]: osdmap e529: 8 total, 8 up, 8 in 2024-04-27T01:22:19.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:19.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:18 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:19.835 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:22:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:22:19.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:19.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-mon[37680]: osdmap e530: 8 total, 8 up, 8 in 2024-04-27T01:22:19.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:22:19.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:22:19.837 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:22:19.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:19.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-mon[26714]: osdmap e530: 8 total, 8 up, 8 in 2024-04-27T01:22:19.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:22:19.837 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:22:19.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:19 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:22:20.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:19 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:20.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:19 smithi181 ceph-mon[30237]: osdmap e530: 8 total, 8 up, 8 in 2024-04-27T01:22:20.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-27T01:22:20.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:22:20.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:19 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-27T01:22:20.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[37680]: pgmap v814: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:20.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:22:20.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:22:20.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[37680]: osdmap e531: 8 total, 8 up, 8 in 2024-04-27T01:22:20.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:22:20.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:20.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[26714]: pgmap v814: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:20.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:22:20.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:22:20.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[26714]: osdmap e531: 8 total, 8 up, 8 in 2024-04-27T01:22:20.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:22:20.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:20 smithi181 ceph-mon[30237]: pgmap v814: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:20 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-27T01:22:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:22:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:20 smithi181 ceph-mon[30237]: osdmap e531: 8 total, 8 up, 8 in 2024-04-27T01:22:21.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:20 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-27T01:22:21.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:21.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:21 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:22:21.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:22:21.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:21 smithi162 ceph-mon[37680]: osdmap e532: 8 total, 8 up, 8 in 2024-04-27T01:22:21.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:21 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:22:21.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:21 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:22:21.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:21 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:22:21.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:22:21.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:21 smithi162 ceph-mon[26714]: osdmap e532: 8 total, 8 up, 8 in 2024-04-27T01:22:21.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:21 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:22:21.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:21 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:22:22.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:21 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-27T01:22:22.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:22:22.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:21 smithi181 ceph-mon[30237]: osdmap e532: 8 total, 8 up, 8 in 2024-04-27T01:22:22.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:21 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-27T01:22:22.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:21 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-27T01:22:22.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[37680]: pgmap v817: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:22.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[37680]: osdmap e533: 8 total, 8 up, 8 in 2024-04-27T01:22:22.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:22.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:22.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-102"}]: dispatch 2024-04-27T01:22:22.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[37680]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-102"}]: dispatch 2024-04-27T01:22:22.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[26714]: pgmap v817: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:22.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[26714]: osdmap e533: 8 total, 8 up, 8 in 2024-04-27T01:22:22.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:22.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:22.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-102"}]: dispatch 2024-04-27T01:22:22.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:22 smithi162 ceph-mon[26714]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-102"}]: dispatch 2024-04-27T01:22:23.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:22:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:22:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:22 smithi181 ceph-mon[30237]: pgmap v817: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 612 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:22 smithi181 ceph-mon[30237]: osdmap e533: 8 total, 8 up, 8 in 2024-04-27T01:22:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:22 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-6"}]: dispatch 2024-04-27T01:22:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1238646027' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-102"}]: dispatch 2024-04-27T01:22:23.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:22 smithi181 ceph-mon[30237]: from='client.28474 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-6", "tierpool": "test-rados-api-smithi162-74551-102"}]: dispatch 2024-04-27T01:22:23.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:23 smithi162 ceph-mon[37680]: osdmap e534: 8 total, 8 up, 8 in 2024-04-27T01:22:23.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:23.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:23 smithi162 ceph-mon[26714]: osdmap e534: 8 total, 8 up, 8 in 2024-04-27T01:22:23.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:24.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:23 smithi181 ceph-mon[30237]: osdmap e534: 8 total, 8 up, 8 in 2024-04-27T01:22:24.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:24 smithi162 ceph-mon[37680]: pgmap v820: 196 pgs: 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:22:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:24 smithi162 ceph-mon[37680]: osdmap e535: 8 total, 8 up, 8 in 2024-04-27T01:22:24.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:24.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:24.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:22:24.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:24 smithi162 ceph-mon[26714]: pgmap v820: 196 pgs: 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:22:24.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:24 smithi162 ceph-mon[26714]: osdmap e535: 8 total, 8 up, 8 in 2024-04-27T01:22:24.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:24.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:24.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:22:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:24 smithi181 ceph-mon[30237]: pgmap v820: 196 pgs: 196 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:22:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:24 smithi181 ceph-mon[30237]: osdmap e535: 8 total, 8 up, 8 in 2024-04-27T01:22:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:22:26.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:22:26.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:25 smithi162 ceph-mon[37680]: osdmap e536: 8 total, 8 up, 8 in 2024-04-27T01:22:26.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-74551-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:26.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:22:26.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:25 smithi162 ceph-mon[26714]: osdmap e536: 8 total, 8 up, 8 in 2024-04-27T01:22:26.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-74551-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:26.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:22:26.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:25 smithi181 ceph-mon[30237]: osdmap e536: 8 total, 8 up, 8 in 2024-04-27T01:22:26.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-74551-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:27.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:26 smithi162 ceph-mon[37680]: pgmap v823: 164 pgs: 164 active+clean; 583 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:27.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:26 smithi162 ceph-mon[37680]: osdmap e537: 8 total, 8 up, 8 in 2024-04-27T01:22:27.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-74551-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-74551-104"}]': finished 2024-04-27T01:22:27.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:26 smithi162 ceph-mon[37680]: osdmap e538: 8 total, 8 up, 8 in 2024-04-27T01:22:27.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:26 smithi162 ceph-mon[26714]: pgmap v823: 164 pgs: 164 active+clean; 583 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:27.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:26 smithi162 ceph-mon[26714]: osdmap e537: 8 total, 8 up, 8 in 2024-04-27T01:22:27.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-74551-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-74551-104"}]': finished 2024-04-27T01:22:27.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:26 smithi162 ceph-mon[26714]: osdmap e538: 8 total, 8 up, 8 in 2024-04-27T01:22:27.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:26 smithi181 ceph-mon[30237]: pgmap v823: 164 pgs: 164 active+clean; 583 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:27.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:26 smithi181 ceph-mon[30237]: osdmap e537: 8 total, 8 up, 8 in 2024-04-27T01:22:27.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi162-74551-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi162-74551-104"}]': finished 2024-04-27T01:22:27.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:26 smithi181 ceph-mon[30237]: osdmap e538: 8 total, 8 up, 8 in 2024-04-27T01:22:28.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:28 smithi162 ceph-mon[37680]: pgmap v826: 172 pgs: 8 creating+peering, 164 active+clean; 583 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:28.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:28 smithi162 ceph-mon[37680]: osdmap e539: 8 total, 8 up, 8 in 2024-04-27T01:22:28.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:28 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:28.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:28 smithi162 ceph-mon[26714]: pgmap v826: 172 pgs: 8 creating+peering, 164 active+clean; 583 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:28.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:28 smithi162 ceph-mon[26714]: osdmap e539: 8 total, 8 up, 8 in 2024-04-27T01:22:28.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:28 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:29.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:28 smithi181 ceph-mon[30237]: pgmap v826: 172 pgs: 8 creating+peering, 164 active+clean; 583 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:29.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:28 smithi181 ceph-mon[30237]: osdmap e539: 8 total, 8 up, 8 in 2024-04-27T01:22:29.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:28 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:30.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:29 smithi162 ceph-mon[26714]: osdmap e540: 8 total, 8 up, 8 in 2024-04-27T01:22:30.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:30.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:30.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:29 smithi162 ceph-mon[26714]: osdmap e541: 8 total, 8 up, 8 in 2024-04-27T01:22:30.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:22:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:22:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:22:30.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:29 smithi162 ceph-mon[37680]: osdmap e540: 8 total, 8 up, 8 in 2024-04-27T01:22:30.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:30.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:30.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:29 smithi162 ceph-mon[37680]: osdmap e541: 8 total, 8 up, 8 in 2024-04-27T01:22:30.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:29 smithi181 ceph-mon[30237]: osdmap e540: 8 total, 8 up, 8 in 2024-04-27T01:22:30.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:30.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:30.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:29 smithi181 ceph-mon[30237]: osdmap e541: 8 total, 8 up, 8 in 2024-04-27T01:22:31.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:30 smithi162 ceph-mon[37680]: pgmap v829: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:31.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:30 smithi162 ceph-mon[37680]: osdmap e542: 8 total, 8 up, 8 in 2024-04-27T01:22:31.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:31.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:30 smithi162 ceph-mon[26714]: pgmap v829: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:31.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:30 smithi162 ceph-mon[26714]: osdmap e542: 8 total, 8 up, 8 in 2024-04-27T01:22:31.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:31.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:30 smithi181 ceph-mon[30237]: pgmap v829: 204 pgs: 32 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:31.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:30 smithi181 ceph-mon[30237]: osdmap e542: 8 total, 8 up, 8 in 2024-04-27T01:22:31.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:32.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:32 smithi162 ceph-mon[37680]: pgmap v832: 236 pgs: 64 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:32.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:32.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:32 smithi162 ceph-mon[37680]: osdmap e543: 8 total, 8 up, 8 in 2024-04-27T01:22:32.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]: dispatch 2024-04-27T01:22:32.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:32 smithi162 ceph-mon[26714]: pgmap v832: 236 pgs: 64 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:32.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:32.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:32 smithi162 ceph-mon[26714]: osdmap e543: 8 total, 8 up, 8 in 2024-04-27T01:22:32.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]: dispatch 2024-04-27T01:22:33.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:22:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:22:33.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:32 smithi181 ceph-mon[30237]: pgmap v832: 236 pgs: 64 unknown, 8 creating+peering, 164 active+clean; 583 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:33.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:33.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:32 smithi181 ceph-mon[30237]: osdmap e543: 8 total, 8 up, 8 in 2024-04-27T01:22:33.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]: dispatch 2024-04-27T01:22:33.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]': finished 2024-04-27T01:22:33.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[37680]: osdmap e544: 8 total, 8 up, 8 in 2024-04-27T01:22:33.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-107", "overlaypool": "test-rados-api-smithi162-74551-107-cache"}]: dispatch 2024-04-27T01:22:33.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:33.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-107", "overlaypool": "test-rados-api-smithi162-74551-107-cache"}]': finished 2024-04-27T01:22:33.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[37680]: osdmap e545: 8 total, 8 up, 8 in 2024-04-27T01:22:33.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-107-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:22:33.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]': finished 2024-04-27T01:22:33.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[26714]: osdmap e544: 8 total, 8 up, 8 in 2024-04-27T01:22:33.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-107", "overlaypool": "test-rados-api-smithi162-74551-107-cache"}]: dispatch 2024-04-27T01:22:33.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:33.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-107", "overlaypool": "test-rados-api-smithi162-74551-107-cache"}]': finished 2024-04-27T01:22:33.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[26714]: osdmap e545: 8 total, 8 up, 8 in 2024-04-27T01:22:33.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-107-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:22:34.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]': finished 2024-04-27T01:22:34.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:33 smithi181 ceph-mon[30237]: osdmap e544: 8 total, 8 up, 8 in 2024-04-27T01:22:34.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-107", "overlaypool": "test-rados-api-smithi162-74551-107-cache"}]: dispatch 2024-04-27T01:22:34.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:34.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-107", "overlaypool": "test-rados-api-smithi162-74551-107-cache"}]': finished 2024-04-27T01:22:34.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:33 smithi181 ceph-mon[30237]: osdmap e545: 8 total, 8 up, 8 in 2024-04-27T01:22:34.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-107-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:22:34.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:34 smithi162 ceph-mon[37680]: pgmap v835: 236 pgs: 236 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:34.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:34 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:34.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:34 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:22:34.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-107-cache", "mode": "writeback"}]': finished 2024-04-27T01:22:34.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:34 smithi162 ceph-mon[37680]: osdmap e546: 8 total, 8 up, 8 in 2024-04-27T01:22:34.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:34 smithi162 ceph-mon[26714]: pgmap v835: 236 pgs: 236 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:34.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:34 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:34.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:34 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:22:34.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-107-cache", "mode": "writeback"}]': finished 2024-04-27T01:22:34.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:34 smithi162 ceph-mon[26714]: osdmap e546: 8 total, 8 up, 8 in 2024-04-27T01:22:35.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:34 smithi181 ceph-mon[30237]: pgmap v835: 236 pgs: 236 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:35.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:34 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:35.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:34 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:22:35.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-107-cache", "mode": "writeback"}]': finished 2024-04-27T01:22:35.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:34 smithi181 ceph-mon[30237]: osdmap e546: 8 total, 8 up, 8 in 2024-04-27T01:22:35.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:35 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-107"}]: dispatch 2024-04-27T01:22:35.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:35 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:22:35.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:35 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:35.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:35 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-107"}]: dispatch 2024-04-27T01:22:35.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:35 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:22:35.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:35 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:36.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:35 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-107"}]: dispatch 2024-04-27T01:22:36.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:35 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:22:36.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:35 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:37.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:36 smithi181 ceph-mon[30237]: pgmap v838: 236 pgs: 236 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:37.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-107"}]': finished 2024-04-27T01:22:37.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:36 smithi181 ceph-mon[30237]: osdmap e547: 8 total, 8 up, 8 in 2024-04-27T01:22:37.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]: dispatch 2024-04-27T01:22:37.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:36 smithi162 ceph-mon[37680]: pgmap v838: 236 pgs: 236 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:37.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-107"}]': finished 2024-04-27T01:22:37.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:36 smithi162 ceph-mon[37680]: osdmap e547: 8 total, 8 up, 8 in 2024-04-27T01:22:37.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]: dispatch 2024-04-27T01:22:37.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:36 smithi162 ceph-mon[26714]: pgmap v838: 236 pgs: 236 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:22:37.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-107"}]': finished 2024-04-27T01:22:37.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:36 smithi162 ceph-mon[26714]: osdmap e547: 8 total, 8 up, 8 in 2024-04-27T01:22:37.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]: dispatch 2024-04-27T01:22:38.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:37 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:22:38.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]': finished 2024-04-27T01:22:38.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:37 smithi181 ceph-mon[30237]: osdmap e548: 8 total, 8 up, 8 in 2024-04-27T01:22:38.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:37 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:22:38.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]': finished 2024-04-27T01:22:38.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:37 smithi162 ceph-mon[26714]: osdmap e548: 8 total, 8 up, 8 in 2024-04-27T01:22:38.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:37 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:22:38.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/192475692' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-107", "tierpool": "test-rados-api-smithi162-74551-107-cache"}]': finished 2024-04-27T01:22:38.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:37 smithi162 ceph-mon[37680]: osdmap e548: 8 total, 8 up, 8 in 2024-04-27T01:22:39.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:38 smithi181 ceph-mon[30237]: pgmap v841: 236 pgs: 236 active+clean; 583 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:22:39.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:38 smithi181 ceph-mon[30237]: osdmap e549: 8 total, 8 up, 8 in 2024-04-27T01:22:39.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:38 smithi162 ceph-mon[37680]: pgmap v841: 236 pgs: 236 active+clean; 583 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:22:39.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:38 smithi162 ceph-mon[37680]: osdmap e549: 8 total, 8 up, 8 in 2024-04-27T01:22:39.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:38 smithi162 ceph-mon[26714]: pgmap v841: 236 pgs: 236 active+clean; 583 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:22:39.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:38 smithi162 ceph-mon[26714]: osdmap e549: 8 total, 8 up, 8 in 2024-04-27T01:22:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:39 smithi181 ceph-mon[30237]: osdmap e550: 8 total, 8 up, 8 in 2024-04-27T01:22:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:39 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:39 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:22:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:39 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:22:40.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:22:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:22:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[37680]: osdmap e550: 8 total, 8 up, 8 in 2024-04-27T01:22:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:22:40.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:22:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[26714]: osdmap e550: 8 total, 8 up, 8 in 2024-04-27T01:22:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:22:40.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:39 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:22:41.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:40 smithi181 ceph-mon[30237]: pgmap v844: 172 pgs: 172 active+clean; 583 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:41.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:40 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:22:41.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:40 smithi181 ceph-mon[30237]: osdmap e551: 8 total, 8 up, 8 in 2024-04-27T01:22:41.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:41.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:40 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:41.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:40 smithi162 ceph-mon[37680]: pgmap v844: 172 pgs: 172 active+clean; 583 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:41.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:40 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:22:41.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:40 smithi162 ceph-mon[37680]: osdmap e551: 8 total, 8 up, 8 in 2024-04-27T01:22:41.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:41.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:40 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:41.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:40 smithi162 ceph-mon[26714]: pgmap v844: 172 pgs: 172 active+clean; 583 KiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:41.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:40 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:22:41.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:40 smithi162 ceph-mon[26714]: osdmap e551: 8 total, 8 up, 8 in 2024-04-27T01:22:41.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:41.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:40 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:42.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:41 smithi181 ceph-mon[30237]: osdmap e552: 8 total, 8 up, 8 in 2024-04-27T01:22:42.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:41 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:42.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:41 smithi181 ceph-mon[30237]: osdmap e553: 8 total, 8 up, 8 in 2024-04-27T01:22:42.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:41 smithi162 ceph-mon[37680]: osdmap e552: 8 total, 8 up, 8 in 2024-04-27T01:22:42.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:41 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:42.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:41 smithi162 ceph-mon[37680]: osdmap e553: 8 total, 8 up, 8 in 2024-04-27T01:22:42.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:41 smithi162 ceph-mon[26714]: osdmap e552: 8 total, 8 up, 8 in 2024-04-27T01:22:42.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:41 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:42.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:41 smithi162 ceph-mon[26714]: osdmap e553: 8 total, 8 up, 8 in 2024-04-27T01:22:43.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:22:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:22:43.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:42 smithi181 ceph-mon[30237]: pgmap v847: 172 pgs: 172 active+clean; 583 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:43.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:42 smithi181 ceph-mon[30237]: osdmap e554: 8 total, 8 up, 8 in 2024-04-27T01:22:43.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:43.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:42 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:43.236 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:42 smithi162 ceph-mon[37680]: pgmap v847: 172 pgs: 172 active+clean; 583 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:43.236 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:42 smithi162 ceph-mon[37680]: osdmap e554: 8 total, 8 up, 8 in 2024-04-27T01:22:43.236 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:43.236 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:42 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:43.237 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:42 smithi162 ceph-mon[26714]: pgmap v847: 172 pgs: 172 active+clean; 583 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:43.237 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:42 smithi162 ceph-mon[26714]: osdmap e554: 8 total, 8 up, 8 in 2024-04-27T01:22:43.237 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:43.237 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:42 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:22:44.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:44.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:43 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:44.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:43 smithi181 ceph-mon[30237]: osdmap e555: 8 total, 8 up, 8 in 2024-04-27T01:22:44.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:44.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:43 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:44.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:44.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:43 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:44.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:43 smithi162 ceph-mon[37680]: osdmap e555: 8 total, 8 up, 8 in 2024-04-27T01:22:44.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:44.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:43 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:44.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:44.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:43 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:22:44.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:43 smithi162 ceph-mon[26714]: osdmap e555: 8 total, 8 up, 8 in 2024-04-27T01:22:44.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:44.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:43 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:45.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:44 smithi181 ceph-mon[30237]: pgmap v850: 212 pgs: 32 unknown, 8 creating+peering, 172 active+clean; 583 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:45.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:44 smithi181 ceph-mon[30237]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:45.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:44 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]': finished 2024-04-27T01:22:45.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:44 smithi181 ceph-mon[30237]: osdmap e556: 8 total, 8 up, 8 in 2024-04-27T01:22:45.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-109", "overlaypool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:45.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:44 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-109", "overlaypool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:45.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[37680]: pgmap v850: 212 pgs: 32 unknown, 8 creating+peering, 172 active+clean; 583 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:45.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[37680]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:45.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]': finished 2024-04-27T01:22:45.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[37680]: osdmap e556: 8 total, 8 up, 8 in 2024-04-27T01:22:45.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-109", "overlaypool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:45.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-109", "overlaypool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:45.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[26714]: pgmap v850: 212 pgs: 32 unknown, 8 creating+peering, 172 active+clean; 583 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:45.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[26714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:45.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]': finished 2024-04-27T01:22:45.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[26714]: osdmap e556: 8 total, 8 up, 8 in 2024-04-27T01:22:45.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-109", "overlaypool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:45.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:44 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-109", "overlaypool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:47.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:46 smithi181 ceph-mon[30237]: pgmap v853: 212 pgs: 32 unknown, 8 creating+peering, 172 active+clean; 583 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:47.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:46 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-109", "overlaypool": "test-rados-api-smithi162-74551-109-cache"}]': finished 2024-04-27T01:22:47.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:46 smithi181 ceph-mon[30237]: osdmap e557: 8 total, 8 up, 8 in 2024-04-27T01:22:47.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-109-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:22:47.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:46 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-109-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:22:47.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:46 smithi162 ceph-mon[37680]: pgmap v853: 212 pgs: 32 unknown, 8 creating+peering, 172 active+clean; 583 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:47.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:46 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-109", "overlaypool": "test-rados-api-smithi162-74551-109-cache"}]': finished 2024-04-27T01:22:47.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:46 smithi162 ceph-mon[37680]: osdmap e557: 8 total, 8 up, 8 in 2024-04-27T01:22:47.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-109-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:22:47.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:46 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-109-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:22:47.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:46 smithi162 ceph-mon[26714]: pgmap v853: 212 pgs: 32 unknown, 8 creating+peering, 172 active+clean; 583 KiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:47.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:46 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-109", "overlaypool": "test-rados-api-smithi162-74551-109-cache"}]': finished 2024-04-27T01:22:47.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:46 smithi162 ceph-mon[26714]: osdmap e557: 8 total, 8 up, 8 in 2024-04-27T01:22:47.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-109-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:22:47.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:46 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-109-cache", "mode": "writeback"}]: dispatch 2024-04-27T01:22:48.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:47 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:22:48.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:47 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-109-cache", "mode": "writeback"}]': finished 2024-04-27T01:22:48.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:47 smithi181 ceph-mon[30237]: osdmap e558: 8 total, 8 up, 8 in 2024-04-27T01:22:48.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:48.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:47 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:48.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:47 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:22:48.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:47 smithi181 ceph-mon[30237]: osdmap e559: 8 total, 8 up, 8 in 2024-04-27T01:22:48.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:48.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:47 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:48.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:22:48.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-109-cache", "mode": "writeback"}]': finished 2024-04-27T01:22:48.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[26714]: osdmap e558: 8 total, 8 up, 8 in 2024-04-27T01:22:48.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:48.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:48.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[26714]: osdmap e559: 8 total, 8 up, 8 in 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-109-cache", "mode": "writeback"}]': finished 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[37680]: osdmap e558: 8 total, 8 up, 8 in 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[37680]: osdmap e559: 8 total, 8 up, 8 in 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:48.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:47 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:22:49.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:48 smithi181 ceph-mon[30237]: pgmap v856: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:49.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:48 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:22:49.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:48 smithi181 ceph-mon[30237]: osdmap e560: 8 total, 8 up, 8 in 2024-04-27T01:22:49.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:49.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:48 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:49.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:48 smithi162 ceph-mon[37680]: pgmap v856: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:49.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:48 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:22:49.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:48 smithi162 ceph-mon[37680]: osdmap e560: 8 total, 8 up, 8 in 2024-04-27T01:22:49.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:49.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:48 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:49.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:48 smithi162 ceph-mon[26714]: pgmap v856: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:49.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:48 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:22:49.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:48 smithi162 ceph-mon[26714]: osdmap e560: 8 total, 8 up, 8 in 2024-04-27T01:22:49.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:49.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:48 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:22:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:49 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:49 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:22:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:49 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:22:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:49 smithi181 ceph-mon[30237]: osdmap e561: 8 total, 8 up, 8 in 2024-04-27T01:22:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-27T01:22:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:49 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-27T01:22:50.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:22:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:22:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:22:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:22:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[37680]: osdmap e561: 8 total, 8 up, 8 in 2024-04-27T01:22:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-27T01:22:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-27T01:22:50.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:22:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:22:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[26714]: osdmap e561: 8 total, 8 up, 8 in 2024-04-27T01:22:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-27T01:22:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:49 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-27T01:22:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:50 smithi181 ceph-mon[30237]: pgmap v859: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:50 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:22:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:50 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-27T01:22:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:50 smithi181 ceph-mon[30237]: osdmap e562: 8 total, 8 up, 8 in 2024-04-27T01:22:51.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:50 smithi162 ceph-mon[37680]: pgmap v859: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:51.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:50 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:22:51.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:51.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:50 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-27T01:22:51.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:50 smithi162 ceph-mon[37680]: osdmap e562: 8 total, 8 up, 8 in 2024-04-27T01:22:51.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:50 smithi162 ceph-mon[26714]: pgmap v859: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:51.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:50 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:22:51.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:22:51.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:50 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-27T01:22:51.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:50 smithi162 ceph-mon[26714]: osdmap e562: 8 total, 8 up, 8 in 2024-04-27T01:22:52.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:52.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:51 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:52.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:52.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:51 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:52.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:52.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:51 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:53.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:22:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:22:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:52 smithi181 ceph-mon[30237]: pgmap v862: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:52 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:52 smithi181 ceph-mon[30237]: osdmap e563: 8 total, 8 up, 8 in 2024-04-27T01:22:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:53.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:52 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:52 smithi162 ceph-mon[37680]: pgmap v862: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:52 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:52 smithi162 ceph-mon[37680]: osdmap e563: 8 total, 8 up, 8 in 2024-04-27T01:22:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:52 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:52 smithi162 ceph-mon[26714]: pgmap v862: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:22:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:52 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:52 smithi162 ceph-mon[26714]: osdmap e563: 8 total, 8 up, 8 in 2024-04-27T01:22:53.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:53.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:52 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]: dispatch 2024-04-27T01:22:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:53 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]': finished 2024-04-27T01:22:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:53 smithi181 ceph-mon[30237]: osdmap e564: 8 total, 8 up, 8 in 2024-04-27T01:22:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:53 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]': finished 2024-04-27T01:22:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:53 smithi162 ceph-mon[37680]: osdmap e564: 8 total, 8 up, 8 in 2024-04-27T01:22:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:53 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-109", "tierpool": "test-rados-api-smithi162-74551-109-cache"}]': finished 2024-04-27T01:22:54.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:53 smithi162 ceph-mon[26714]: osdmap e564: 8 total, 8 up, 8 in 2024-04-27T01:22:54.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:22:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:54 smithi181 ceph-mon[30237]: pgmap v865: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:22:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:54 smithi181 ceph-mon[30237]: osdmap e565: 8 total, 8 up, 8 in 2024-04-27T01:22:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:54 smithi181 ceph-mon[30237]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:55.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:54 smithi162 ceph-mon[37680]: pgmap v865: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:22:55.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:54 smithi162 ceph-mon[37680]: osdmap e565: 8 total, 8 up, 8 in 2024-04-27T01:22:55.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:54 smithi162 ceph-mon[37680]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:54 smithi162 ceph-mon[26714]: pgmap v865: 212 pgs: 212 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-27T01:22:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:54 smithi162 ceph-mon[26714]: osdmap e565: 8 total, 8 up, 8 in 2024-04-27T01:22:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:54 smithi162 ceph-mon[26714]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:22:56.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:55 smithi181 ceph-mon[30237]: osdmap e566: 8 total, 8 up, 8 in 2024-04-27T01:22:56.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:56.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:55 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:56.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:55 smithi162 ceph-mon[37680]: osdmap e566: 8 total, 8 up, 8 in 2024-04-27T01:22:56.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:56.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:55 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:56.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:55 smithi162 ceph-mon[26714]: osdmap e566: 8 total, 8 up, 8 in 2024-04-27T01:22:56.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:56.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:55 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:57.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:56 smithi181 ceph-mon[30237]: pgmap v868: 172 pgs: 172 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:57.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:56 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:57.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:57.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:56 smithi181 ceph-mon[30237]: osdmap e567: 8 total, 8 up, 8 in 2024-04-27T01:22:57.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:56 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:57.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:56 smithi162 ceph-mon[37680]: pgmap v868: 172 pgs: 172 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:57.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:56 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:57.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:57.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:56 smithi162 ceph-mon[37680]: osdmap e567: 8 total, 8 up, 8 in 2024-04-27T01:22:57.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:56 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:57.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:56 smithi162 ceph-mon[26714]: pgmap v868: 172 pgs: 172 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:22:57.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:56 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:57.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1480253927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:57.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:56 smithi162 ceph-mon[26714]: osdmap e567: 8 total, 8 up, 8 in 2024-04-27T01:22:57.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:56 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]: dispatch 2024-04-27T01:22:58.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:57 smithi181 ceph-mon[30237]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:58.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:57 smithi181 ceph-mon[30237]: osdmap e568: 8 total, 8 up, 8 in 2024-04-27T01:22:58.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:57 smithi162 ceph-mon[37680]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:58.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:57 smithi162 ceph-mon[37680]: osdmap e568: 8 total, 8 up, 8 in 2024-04-27T01:22:58.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:57 smithi162 ceph-mon[26714]: from='client.57053 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-109"}]': finished 2024-04-27T01:22:58.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:57 smithi162 ceph-mon[26714]: osdmap e568: 8 total, 8 up, 8 in 2024-04-27T01:22:59.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:58 smithi181 ceph-mon[30237]: pgmap v871: 172 pgs: 172 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-27T01:22:59.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:58 smithi181 ceph-mon[30237]: osdmap e569: 8 total, 8 up, 8 in 2024-04-27T01:22:59.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:59.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:58 smithi162 ceph-mon[37680]: pgmap v871: 172 pgs: 172 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-27T01:22:59.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:58 smithi162 ceph-mon[37680]: osdmap e569: 8 total, 8 up, 8 in 2024-04-27T01:22:59.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:22:59.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:58 smithi162 ceph-mon[26714]: pgmap v871: 172 pgs: 172 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-27T01:22:59.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:58 smithi162 ceph-mon[26714]: osdmap e569: 8 total, 8 up, 8 in 2024-04-27T01:22:59.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:23:00.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104"}]': finished 2024-04-27T01:23:00.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:59 smithi181 ceph-mon[30237]: osdmap e570: 8 total, 8 up, 8 in 2024-04-27T01:23:00.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:23:00.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:59 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:00.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:22:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-74551-104"}]': finished 2024-04-27T01:23:00.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:22:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:23:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104"}]': finished 2024-04-27T01:23:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-mon[26714]: osdmap e570: 8 total, 8 up, 8 in 2024-04-27T01:23:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:23:00.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:00.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-74551-104"}]': finished 2024-04-27T01:23:00.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi162-74551-104"}]': finished 2024-04-27T01:23:00.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-mon[37680]: osdmap e570: 8 total, 8 up, 8 in 2024-04-27T01:23:00.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-74551-104"}]: dispatch 2024-04-27T01:23:00.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:00.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:22:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2950583284' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi162-74551-104"}]': finished 2024-04-27T01:23:01.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:00 smithi181 ceph-mon[30237]: pgmap v874: 164 pgs: 164 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-27T01:23:01.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:00 smithi181 ceph-mon[30237]: osdmap e571: 8 total, 8 up, 8 in 2024-04-27T01:23:01.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:01.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:01.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:23:01.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:00 smithi162 ceph-mon[37680]: pgmap v874: 164 pgs: 164 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-27T01:23:01.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:00 smithi162 ceph-mon[37680]: osdmap e571: 8 total, 8 up, 8 in 2024-04-27T01:23:01.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:01.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:01.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:23:01.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:00 smithi162 ceph-mon[26714]: pgmap v874: 164 pgs: 164 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-27T01:23:01.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:00 smithi162 ceph-mon[26714]: osdmap e571: 8 total, 8 up, 8 in 2024-04-27T01:23:01.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:01.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:01.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-27T01:23:02.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:23:02.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:01 smithi181 ceph-mon[30237]: osdmap e572: 8 total, 8 up, 8 in 2024-04-27T01:23:02.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:02.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:23:02.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:01 smithi162 ceph-mon[26714]: osdmap e572: 8 total, 8 up, 8 in 2024-04-27T01:23:02.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:02.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi162-74551-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-27T01:23:02.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:01 smithi162 ceph-mon[37680]: osdmap e572: 8 total, 8 up, 8 in 2024-04-27T01:23:02.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:03.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:23:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:23:03.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:02 smithi181 ceph-mon[30237]: pgmap v877: 164 pgs: 164 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:03.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:02 smithi181 ceph-mon[30237]: osdmap e573: 8 total, 8 up, 8 in 2024-04-27T01:23:03.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:02 smithi162 ceph-mon[37680]: pgmap v877: 164 pgs: 164 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:03.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:02 smithi162 ceph-mon[37680]: osdmap e573: 8 total, 8 up, 8 in 2024-04-27T01:23:03.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:02 smithi162 ceph-mon[26714]: pgmap v877: 164 pgs: 164 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:03.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:02 smithi162 ceph-mon[26714]: osdmap e573: 8 total, 8 up, 8 in 2024-04-27T01:23:04.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:04.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:04.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:03 smithi181 ceph-mon[30237]: osdmap e574: 8 total, 8 up, 8 in 2024-04-27T01:23:04.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:04.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:04.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:03 smithi162 ceph-mon[37680]: osdmap e574: 8 total, 8 up, 8 in 2024-04-27T01:23:04.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:04.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi162-74551-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:04.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:03 smithi162 ceph-mon[26714]: osdmap e574: 8 total, 8 up, 8 in 2024-04-27T01:23:05.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:04 smithi162 ceph-mon[37680]: pgmap v880: 172 pgs: 8 unknown, 164 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:05.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:04 smithi162 ceph-mon[37680]: osdmap e575: 8 total, 8 up, 8 in 2024-04-27T01:23:05.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:05.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:04 smithi162 ceph-mon[26714]: pgmap v880: 172 pgs: 8 unknown, 164 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:05.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:04 smithi162 ceph-mon[26714]: osdmap e575: 8 total, 8 up, 8 in 2024-04-27T01:23:05.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:05.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:04 smithi181 ceph-mon[30237]: pgmap v880: 172 pgs: 8 unknown, 164 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:05.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:04 smithi181 ceph-mon[30237]: osdmap e575: 8 total, 8 up, 8 in 2024-04-27T01:23:05.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:06.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:06.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:05 smithi162 ceph-mon[37680]: osdmap e576: 8 total, 8 up, 8 in 2024-04-27T01:23:06.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:06.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:05 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:06.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:06.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:05 smithi162 ceph-mon[26714]: osdmap e576: 8 total, 8 up, 8 in 2024-04-27T01:23:06.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:06.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:05 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:06.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:06.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:05 smithi181 ceph-mon[30237]: osdmap e576: 8 total, 8 up, 8 in 2024-04-27T01:23:06.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:06.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:05 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:07.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:06 smithi162 ceph-mon[26714]: pgmap v883: 204 pgs: 40 unknown, 164 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:07.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:07.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:06 smithi162 ceph-mon[26714]: osdmap e577: 8 total, 8 up, 8 in 2024-04-27T01:23:07.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-112"}]: dispatch 2024-04-27T01:23:07.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:06 smithi162 ceph-mon[37680]: pgmap v883: 204 pgs: 40 unknown, 164 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:07.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:07.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:06 smithi162 ceph-mon[37680]: osdmap e577: 8 total, 8 up, 8 in 2024-04-27T01:23:07.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-112"}]: dispatch 2024-04-27T01:23:07.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:06 smithi181 ceph-mon[30237]: pgmap v883: 204 pgs: 40 unknown, 164 active+clean; 583 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:07.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:07.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:06 smithi181 ceph-mon[30237]: osdmap e577: 8 total, 8 up, 8 in 2024-04-27T01:23:07.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-112"}]: dispatch 2024-04-27T01:23:08.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-112"}]': finished 2024-04-27T01:23:08.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:07 smithi162 ceph-mon[26714]: osdmap e578: 8 total, 8 up, 8 in 2024-04-27T01:23:08.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:08.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-112"}]': finished 2024-04-27T01:23:08.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:07 smithi162 ceph-mon[37680]: osdmap e578: 8 total, 8 up, 8 in 2024-04-27T01:23:08.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:08.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-112"}]': finished 2024-04-27T01:23:08.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:07 smithi181 ceph-mon[30237]: osdmap e578: 8 total, 8 up, 8 in 2024-04-27T01:23:08.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:08 smithi162 ceph-mon[26714]: pgmap v886: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:23:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:08 smithi162 ceph-mon[26714]: osdmap e579: 8 total, 8 up, 8 in 2024-04-27T01:23:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112"}]: dispatch 2024-04-27T01:23:09.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:08 smithi162 ceph-mon[37680]: pgmap v886: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:23:09.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:09.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:08 smithi162 ceph-mon[37680]: osdmap e579: 8 total, 8 up, 8 in 2024-04-27T01:23:09.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112"}]: dispatch 2024-04-27T01:23:09.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:08 smithi181 ceph-mon[30237]: pgmap v886: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:23:09.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:09.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:08 smithi181 ceph-mon[30237]: osdmap e579: 8 total, 8 up, 8 in 2024-04-27T01:23:09.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112"}]: dispatch 2024-04-27T01:23:10.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112"}]': finished 2024-04-27T01:23:10.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:09 smithi162 ceph-mon[37680]: osdmap e580: 8 total, 8 up, 8 in 2024-04-27T01:23:10.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112"}]': finished 2024-04-27T01:23:10.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:09 smithi162 ceph-mon[26714]: osdmap e580: 8 total, 8 up, 8 in 2024-04-27T01:23:10.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:23:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:23:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:23:10.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-112"}]': finished 2024-04-27T01:23:10.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:09 smithi181 ceph-mon[30237]: osdmap e580: 8 total, 8 up, 8 in 2024-04-27T01:23:11.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:10 smithi162 ceph-mon[37680]: pgmap v889: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:23:11.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:10 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:11.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:10 smithi162 ceph-mon[37680]: osdmap e581: 8 total, 8 up, 8 in 2024-04-27T01:23:11.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:10 smithi162 ceph-mon[26714]: pgmap v889: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:23:11.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:10 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:11.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:10 smithi162 ceph-mon[26714]: osdmap e581: 8 total, 8 up, 8 in 2024-04-27T01:23:11.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:10 smithi181 ceph-mon[30237]: pgmap v889: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:23:11.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:10 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:11.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:10 smithi181 ceph-mon[30237]: osdmap e581: 8 total, 8 up, 8 in 2024-04-27T01:23:12.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:11 smithi162 ceph-mon[37680]: osdmap e582: 8 total, 8 up, 8 in 2024-04-27T01:23:12.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:12.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:11 smithi162 ceph-mon[26714]: osdmap e582: 8 total, 8 up, 8 in 2024-04-27T01:23:12.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:12.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:11 smithi181 ceph-mon[30237]: osdmap e582: 8 total, 8 up, 8 in 2024-04-27T01:23:12.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:13.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:23:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:23:13.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:12 smithi181 ceph-mon[30237]: pgmap v892: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:13.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:13.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:12 smithi181 ceph-mon[30237]: osdmap e583: 8 total, 8 up, 8 in 2024-04-27T01:23:13.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:13.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:12 smithi162 ceph-mon[37680]: pgmap v892: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:13.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:13.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:12 smithi162 ceph-mon[37680]: osdmap e583: 8 total, 8 up, 8 in 2024-04-27T01:23:13.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:13.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:12 smithi162 ceph-mon[26714]: pgmap v892: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:13.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:13.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:12 smithi162 ceph-mon[26714]: osdmap e583: 8 total, 8 up, 8 in 2024-04-27T01:23:13.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:14.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:14.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:14.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:13 smithi181 ceph-mon[30237]: osdmap e584: 8 total, 8 up, 8 in 2024-04-27T01:23:14.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-114"}]: dispatch 2024-04-27T01:23:14.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:14.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:14.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:13 smithi162 ceph-mon[26714]: osdmap e584: 8 total, 8 up, 8 in 2024-04-27T01:23:14.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-114"}]: dispatch 2024-04-27T01:23:14.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:14.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:14.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:13 smithi162 ceph-mon[37680]: osdmap e584: 8 total, 8 up, 8 in 2024-04-27T01:23:14.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-114"}]: dispatch 2024-04-27T01:23:15.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:14 smithi181 ceph-mon[30237]: pgmap v895: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-27T01:23:15.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-114"}]': finished 2024-04-27T01:23:15.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:14 smithi181 ceph-mon[30237]: osdmap e585: 8 total, 8 up, 8 in 2024-04-27T01:23:15.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-114", "mode": "writeback"}]: dispatch 2024-04-27T01:23:15.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:14 smithi162 ceph-mon[37680]: pgmap v895: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-27T01:23:15.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-114"}]': finished 2024-04-27T01:23:15.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:14 smithi162 ceph-mon[37680]: osdmap e585: 8 total, 8 up, 8 in 2024-04-27T01:23:15.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-114", "mode": "writeback"}]: dispatch 2024-04-27T01:23:15.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:14 smithi162 ceph-mon[26714]: pgmap v895: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-27T01:23:15.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-114"}]': finished 2024-04-27T01:23:15.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:14 smithi162 ceph-mon[26714]: osdmap e585: 8 total, 8 up, 8 in 2024-04-27T01:23:15.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-114", "mode": "writeback"}]: dispatch 2024-04-27T01:23:16.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:15 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:16.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-114", "mode": "writeback"}]': finished 2024-04-27T01:23:16.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:15 smithi181 ceph-mon[30237]: osdmap e586: 8 total, 8 up, 8 in 2024-04-27T01:23:16.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:16.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:15 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:16.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-114", "mode": "writeback"}]': finished 2024-04-27T01:23:16.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:15 smithi162 ceph-mon[37680]: osdmap e586: 8 total, 8 up, 8 in 2024-04-27T01:23:16.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:16.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:15 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:16.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-114", "mode": "writeback"}]': finished 2024-04-27T01:23:16.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:15 smithi162 ceph-mon[26714]: osdmap e586: 8 total, 8 up, 8 in 2024-04-27T01:23:16.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:17.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:16 smithi181 ceph-mon[30237]: pgmap v898: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:23:17.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:17.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:16 smithi181 ceph-mon[30237]: osdmap e587: 8 total, 8 up, 8 in 2024-04-27T01:23:17.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114"}]: dispatch 2024-04-27T01:23:17.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:16 smithi162 ceph-mon[26714]: pgmap v898: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:23:17.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:17.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:16 smithi162 ceph-mon[26714]: osdmap e587: 8 total, 8 up, 8 in 2024-04-27T01:23:17.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114"}]: dispatch 2024-04-27T01:23:17.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:16 smithi162 ceph-mon[37680]: pgmap v898: 204 pgs: 204 active+clean; 583 KiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:23:17.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:17.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:16 smithi162 ceph-mon[37680]: osdmap e587: 8 total, 8 up, 8 in 2024-04-27T01:23:17.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114"}]: dispatch 2024-04-27T01:23:18.127 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:17 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:23:18.127 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114"}]': finished 2024-04-27T01:23:18.127 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:17 smithi162 ceph-mon[26714]: osdmap e588: 8 total, 8 up, 8 in 2024-04-27T01:23:18.128 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:17 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:23:18.128 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:17 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:23:18.128 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114"}]': finished 2024-04-27T01:23:18.128 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:17 smithi162 ceph-mon[37680]: osdmap e588: 8 total, 8 up, 8 in 2024-04-27T01:23:18.128 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:17 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:23:18.247 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:17 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:23:18.247 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-114"}]': finished 2024-04-27T01:23:18.247 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:17 smithi181 ceph-mon[30237]: osdmap e588: 8 total, 8 up, 8 in 2024-04-27T01:23:18.247 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:17 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:23:19.160 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:18 smithi162 ceph-mon[37680]: pgmap v901: 204 pgs: 204 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:23:19.161 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:18 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:19.161 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:18 smithi162 ceph-mon[37680]: osdmap e589: 8 total, 8 up, 8 in 2024-04-27T01:23:19.161 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:18 smithi162 ceph-mon[26714]: pgmap v901: 204 pgs: 204 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:23:19.161 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:18 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:19.161 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:18 smithi162 ceph-mon[26714]: osdmap e589: 8 total, 8 up, 8 in 2024-04-27T01:23:19.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:18 smithi181 ceph-mon[30237]: pgmap v901: 204 pgs: 204 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:23:19.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:18 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:19.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:18 smithi181 ceph-mon[30237]: osdmap e589: 8 total, 8 up, 8 in 2024-04-27T01:23:19.944 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:19 smithi181 ceph-mon[30237]: osdmap e590: 8 total, 8 up, 8 in 2024-04-27T01:23:19.945 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:20.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:19 smithi162 ceph-mon[37680]: osdmap e590: 8 total, 8 up, 8 in 2024-04-27T01:23:20.036 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:20.036 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:23:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:23:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:23:20.036 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:19 smithi162 ceph-mon[26714]: osdmap e590: 8 total, 8 up, 8 in 2024-04-27T01:23:20.036 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:20 smithi181 ceph-mon[30237]: pgmap v904: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:20 smithi181 ceph-mon[30237]: osdmap e591: 8 total, 8 up, 8 in 2024-04-27T01:23:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:20 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:20 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:21.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[37680]: pgmap v904: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:21.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:21.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[37680]: osdmap e591: 8 total, 8 up, 8 in 2024-04-27T01:23:21.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:21.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:21.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:21.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[26714]: pgmap v904: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:21.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:21.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[26714]: osdmap e591: 8 total, 8 up, 8 in 2024-04-27T01:23:21.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:21.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:21.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:22.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:21 smithi181 ceph-mon[30237]: osdmap e592: 8 total, 8 up, 8 in 2024-04-27T01:23:22.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:22.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:21 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:22.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:21 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:22.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:21 smithi162 ceph-mon[37680]: osdmap e592: 8 total, 8 up, 8 in 2024-04-27T01:23:22.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:22.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:21 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:22.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:21 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:22.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:21 smithi162 ceph-mon[26714]: osdmap e592: 8 total, 8 up, 8 in 2024-04-27T01:23:22.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:22.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:21 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:22.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:21 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:23.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:23:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:23:23.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:22 smithi181 ceph-mon[30237]: pgmap v907: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:23.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:23.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:22 smithi181 ceph-mon[30237]: osdmap e593: 8 total, 8 up, 8 in 2024-04-27T01:23:23.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-116"}]: dispatch 2024-04-27T01:23:23.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:22 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:23:23.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:22 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:23:23.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:22 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:23.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[26714]: pgmap v907: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:23.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:23.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[26714]: osdmap e593: 8 total, 8 up, 8 in 2024-04-27T01:23:23.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-116"}]: dispatch 2024-04-27T01:23:23.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:23:23.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:23:23.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:23.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[37680]: pgmap v907: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:23.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:23.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[37680]: osdmap e593: 8 total, 8 up, 8 in 2024-04-27T01:23:23.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-116"}]: dispatch 2024-04-27T01:23:23.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:23:23.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:23:23.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:22 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:24.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:24.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-116"}]': finished 2024-04-27T01:23:24.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:23 smithi181 ceph-mon[30237]: osdmap e594: 8 total, 8 up, 8 in 2024-04-27T01:23:24.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-116", "mode": "writeback"}]: dispatch 2024-04-27T01:23:24.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:23 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:24.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-116", "mode": "writeback"}]': finished 2024-04-27T01:23:24.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:23 smithi181 ceph-mon[30237]: osdmap e595: 8 total, 8 up, 8 in 2024-04-27T01:23:24.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:24.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-116"}]': finished 2024-04-27T01:23:24.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[37680]: osdmap e594: 8 total, 8 up, 8 in 2024-04-27T01:23:24.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-116", "mode": "writeback"}]: dispatch 2024-04-27T01:23:24.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:24.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-116", "mode": "writeback"}]': finished 2024-04-27T01:23:24.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[37680]: osdmap e595: 8 total, 8 up, 8 in 2024-04-27T01:23:24.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:24.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-116"}]': finished 2024-04-27T01:23:24.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[26714]: osdmap e594: 8 total, 8 up, 8 in 2024-04-27T01:23:24.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-116", "mode": "writeback"}]: dispatch 2024-04-27T01:23:24.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:24.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-116", "mode": "writeback"}]': finished 2024-04-27T01:23:24.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:23 smithi162 ceph-mon[26714]: osdmap e595: 8 total, 8 up, 8 in 2024-04-27T01:23:25.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:24 smithi181 ceph-mon[30237]: pgmap v910: 204 pgs: 204 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:23:25.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-27T01:23:25.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:24 smithi181 ceph-mon[30237]: from='mon.0 v1:172.21.15.162:0/747439281' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-27T01:23:25.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:24 smithi181 ceph-mon[30237]: 326.6 scrub starts 2024-04-27T01:23:25.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:24 smithi181 ceph-mon[30237]: 326.6 scrub ok 2024-04-27T01:23:25.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:24 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:25.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[37680]: pgmap v910: 204 pgs: 204 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:23:25.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-27T01:23:25.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[37680]: from='mon.0 v1:172.21.15.162:0/747439281' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-27T01:23:25.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[37680]: 326.6 scrub starts 2024-04-27T01:23:25.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[37680]: 326.6 scrub ok 2024-04-27T01:23:25.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:25.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[26714]: pgmap v910: 204 pgs: 204 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:23:25.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-27T01:23:25.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[26714]: from='mon.0 v1:172.21.15.162:0/747439281' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.6"}]: dispatch 2024-04-27T01:23:25.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[26714]: 326.6 scrub starts 2024-04-27T01:23:25.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[26714]: 326.6 scrub ok 2024-04-27T01:23:25.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:24 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:23:27.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:26 smithi181 ceph-mon[30237]: pgmap v912: 204 pgs: 204 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-27T01:23:27.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:26 smithi162 ceph-mon[26714]: pgmap v912: 204 pgs: 204 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-27T01:23:27.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:26 smithi162 ceph-mon[37680]: pgmap v912: 204 pgs: 204 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-27T01:23:29.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:28 smithi181 ceph-mon[30237]: pgmap v913: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:23:29.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:28 smithi162 ceph-mon[26714]: pgmap v913: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:23:29.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:28 smithi162 ceph-mon[37680]: pgmap v913: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:23:30.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:23:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:23:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:23:31.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:30 smithi181 ceph-mon[30237]: pgmap v914: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 965 B/s wr, 2 op/s 2024-04-27T01:23:31.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:30 smithi162 ceph-mon[37680]: pgmap v914: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 965 B/s wr, 2 op/s 2024-04-27T01:23:31.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:30 smithi162 ceph-mon[26714]: pgmap v914: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 965 B/s wr, 2 op/s 2024-04-27T01:23:33.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:23:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:23:33.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:32 smithi181 ceph-mon[30237]: pgmap v915: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 1 op/s 2024-04-27T01:23:33.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:32 smithi162 ceph-mon[37680]: pgmap v915: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 1 op/s 2024-04-27T01:23:33.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:32 smithi162 ceph-mon[26714]: pgmap v915: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 1 op/s 2024-04-27T01:23:34.211 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:34.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:34.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:35.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:34 smithi181 ceph-mon[30237]: pgmap v916: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:35.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:34 smithi162 ceph-mon[37680]: pgmap v916: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:35.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:34 smithi162 ceph-mon[26714]: pgmap v916: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:36 smithi181 ceph-mon[30237]: pgmap v917: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-27T01:23:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:36.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:36 smithi162 ceph-mon[37680]: pgmap v917: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-27T01:23:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:36.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:36 smithi162 ceph-mon[26714]: pgmap v917: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-27T01:23:36.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:23:36.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:38.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:38 smithi181 ceph-mon[30237]: pgmap v918: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:23:38.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:38 smithi162 ceph-mon[37680]: pgmap v918: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:23:38.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:38 smithi162 ceph-mon[26714]: pgmap v918: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:23:40.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:23:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:23:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:23:40.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:40 smithi181 ceph-mon[30237]: pgmap v919: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:23:40.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:40 smithi181 ceph-mon[30237]: osdmap e596: 8 total, 8 up, 8 in 2024-04-27T01:23:40.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:40.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:40 smithi162 ceph-mon[26714]: pgmap v919: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:23:40.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:40 smithi162 ceph-mon[26714]: osdmap e596: 8 total, 8 up, 8 in 2024-04-27T01:23:40.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:40.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:40 smithi162 ceph-mon[37680]: pgmap v919: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:23:40.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:40 smithi162 ceph-mon[37680]: osdmap e596: 8 total, 8 up, 8 in 2024-04-27T01:23:40.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:41.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:41.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:41 smithi181 ceph-mon[30237]: osdmap e597: 8 total, 8 up, 8 in 2024-04-27T01:23:41.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116"}]: dispatch 2024-04-27T01:23:41.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:41 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:23:41.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116"}]': finished 2024-04-27T01:23:41.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:41.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:41 smithi162 ceph-mon[37680]: osdmap e597: 8 total, 8 up, 8 in 2024-04-27T01:23:41.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116"}]: dispatch 2024-04-27T01:23:41.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:41 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:23:41.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116"}]': finished 2024-04-27T01:23:41.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:41.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:41 smithi162 ceph-mon[26714]: osdmap e597: 8 total, 8 up, 8 in 2024-04-27T01:23:41.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116"}]: dispatch 2024-04-27T01:23:41.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:41 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:23:41.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-116"}]': finished 2024-04-27T01:23:42.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:42 smithi181 ceph-mon[30237]: pgmap v922: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-27T01:23:42.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:42 smithi181 ceph-mon[30237]: osdmap e598: 8 total, 8 up, 8 in 2024-04-27T01:23:42.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:42 smithi162 ceph-mon[37680]: pgmap v922: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-27T01:23:42.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:42 smithi162 ceph-mon[37680]: osdmap e598: 8 total, 8 up, 8 in 2024-04-27T01:23:42.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:42 smithi162 ceph-mon[26714]: pgmap v922: 204 pgs: 204 active+clean; 583 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-27T01:23:42.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:42 smithi162 ceph-mon[26714]: osdmap e598: 8 total, 8 up, 8 in 2024-04-27T01:23:43.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:23:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:23:43.709 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:43 smithi181 ceph-mon[30237]: osdmap e599: 8 total, 8 up, 8 in 2024-04-27T01:23:43.710 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:43.736 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:43 smithi162 ceph-mon[37680]: osdmap e599: 8 total, 8 up, 8 in 2024-04-27T01:23:43.736 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:43.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:43 smithi162 ceph-mon[26714]: osdmap e599: 8 total, 8 up, 8 in 2024-04-27T01:23:43.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:44.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:44 smithi181 ceph-mon[30237]: pgmap v925: 172 pgs: 172 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-04-27T01:23:44.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:44 smithi181 ceph-mon[30237]: osdmap e600: 8 total, 8 up, 8 in 2024-04-27T01:23:44.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:44.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:44.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:44 smithi181 ceph-mon[30237]: osdmap e601: 8 total, 8 up, 8 in 2024-04-27T01:23:44.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:44 smithi162 ceph-mon[26714]: pgmap v925: 172 pgs: 172 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-04-27T01:23:44.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:44 smithi162 ceph-mon[26714]: osdmap e600: 8 total, 8 up, 8 in 2024-04-27T01:23:44.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:44.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:44.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:44 smithi162 ceph-mon[26714]: osdmap e601: 8 total, 8 up, 8 in 2024-04-27T01:23:44.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:44 smithi162 ceph-mon[37680]: pgmap v925: 172 pgs: 172 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-04-27T01:23:44.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:44 smithi162 ceph-mon[37680]: osdmap e600: 8 total, 8 up, 8 in 2024-04-27T01:23:44.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:44.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:44.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:44 smithi162 ceph-mon[37680]: osdmap e601: 8 total, 8 up, 8 in 2024-04-27T01:23:46.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:46 smithi162 ceph-mon[26714]: osdmap e602: 8 total, 8 up, 8 in 2024-04-27T01:23:46.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:46.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:46 smithi162 ceph-mon[26714]: pgmap v929: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 786 B/s rd, 1 op/s 2024-04-27T01:23:46.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:46 smithi162 ceph-mon[37680]: osdmap e602: 8 total, 8 up, 8 in 2024-04-27T01:23:46.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:46.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:46 smithi162 ceph-mon[37680]: pgmap v929: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 786 B/s rd, 1 op/s 2024-04-27T01:23:46.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:46 smithi181 ceph-mon[30237]: osdmap e602: 8 total, 8 up, 8 in 2024-04-27T01:23:46.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:46.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:46 smithi181 ceph-mon[30237]: pgmap v929: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 786 B/s rd, 1 op/s 2024-04-27T01:23:47.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:47.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:47 smithi162 ceph-mon[37680]: osdmap e603: 8 total, 8 up, 8 in 2024-04-27T01:23:47.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-118"}]: dispatch 2024-04-27T01:23:47.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:47.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:47 smithi162 ceph-mon[26714]: osdmap e603: 8 total, 8 up, 8 in 2024-04-27T01:23:47.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-118"}]: dispatch 2024-04-27T01:23:47.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:47.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:47 smithi181 ceph-mon[30237]: osdmap e603: 8 total, 8 up, 8 in 2024-04-27T01:23:47.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-118"}]: dispatch 2024-04-27T01:23:48.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-118"}]': finished 2024-04-27T01:23:48.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[37680]: osdmap e604: 8 total, 8 up, 8 in 2024-04-27T01:23:48.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-118", "mode": "writeback"}]: dispatch 2024-04-27T01:23:48.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[37680]: pgmap v932: 204 pgs: 204 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 527 B/s wr, 1 op/s 2024-04-27T01:23:48.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:48.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-118", "mode": "writeback"}]': finished 2024-04-27T01:23:48.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-118"}]': finished 2024-04-27T01:23:48.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[26714]: osdmap e604: 8 total, 8 up, 8 in 2024-04-27T01:23:48.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-118", "mode": "writeback"}]: dispatch 2024-04-27T01:23:48.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[26714]: pgmap v932: 204 pgs: 204 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 527 B/s wr, 1 op/s 2024-04-27T01:23:48.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:48.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-118", "mode": "writeback"}]': finished 2024-04-27T01:23:48.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-118"}]': finished 2024-04-27T01:23:48.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:48 smithi181 ceph-mon[30237]: osdmap e604: 8 total, 8 up, 8 in 2024-04-27T01:23:48.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-118", "mode": "writeback"}]: dispatch 2024-04-27T01:23:48.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:48 smithi181 ceph-mon[30237]: pgmap v932: 204 pgs: 204 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 527 B/s wr, 1 op/s 2024-04-27T01:23:48.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:48 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:48.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-118", "mode": "writeback"}]': finished 2024-04-27T01:23:49.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:49 smithi162 ceph-mon[37680]: osdmap e605: 8 total, 8 up, 8 in 2024-04-27T01:23:49.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:49 smithi162 ceph-mon[37680]: osdmap e606: 8 total, 8 up, 8 in 2024-04-27T01:23:49.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:49 smithi162 ceph-mon[26714]: osdmap e605: 8 total, 8 up, 8 in 2024-04-27T01:23:49.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:49 smithi162 ceph-mon[26714]: osdmap e606: 8 total, 8 up, 8 in 2024-04-27T01:23:49.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:49 smithi181 ceph-mon[30237]: osdmap e605: 8 total, 8 up, 8 in 2024-04-27T01:23:49.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:49 smithi181 ceph-mon[30237]: osdmap e606: 8 total, 8 up, 8 in 2024-04-27T01:23:50.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:23:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:23:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:23:50.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:50 smithi162 ceph-mon[37680]: pgmap v935: 204 pgs: 204 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:23:50.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:50.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:50 smithi162 ceph-mon[26714]: pgmap v935: 204 pgs: 204 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:23:50.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:50.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:50 smithi181 ceph-mon[30237]: pgmap v935: 204 pgs: 204 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-27T01:23:50.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:51.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:51.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:51 smithi181 ceph-mon[30237]: osdmap e607: 8 total, 8 up, 8 in 2024-04-27T01:23:51.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118"}]: dispatch 2024-04-27T01:23:51.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:51 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:51.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:51.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:51 smithi162 ceph-mon[37680]: osdmap e607: 8 total, 8 up, 8 in 2024-04-27T01:23:51.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118"}]: dispatch 2024-04-27T01:23:51.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:51 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:51.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:51.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:51 smithi162 ceph-mon[26714]: osdmap e607: 8 total, 8 up, 8 in 2024-04-27T01:23:51.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118"}]: dispatch 2024-04-27T01:23:51.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:51 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:23:53.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:23:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:23:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:52 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:23:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118"}]': finished 2024-04-27T01:23:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:52 smithi181 ceph-mon[30237]: pgmap v937: 204 pgs: 204 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:52 smithi181 ceph-mon[30237]: osdmap e608: 8 total, 8 up, 8 in 2024-04-27T01:23:53.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:52 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:23:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118"}]': finished 2024-04-27T01:23:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:52 smithi162 ceph-mon[37680]: pgmap v937: 204 pgs: 204 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:52 smithi162 ceph-mon[37680]: osdmap e608: 8 total, 8 up, 8 in 2024-04-27T01:23:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:52 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:23:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-118"}]': finished 2024-04-27T01:23:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:52 smithi162 ceph-mon[26714]: pgmap v937: 204 pgs: 204 active+clean; 583 KiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:23:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:52 smithi162 ceph-mon[26714]: osdmap e608: 8 total, 8 up, 8 in 2024-04-27T01:23:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:53 smithi181 ceph-mon[30237]: osdmap e609: 8 total, 8 up, 8 in 2024-04-27T01:23:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:53 smithi181 ceph-mon[30237]: osdmap e610: 8 total, 8 up, 8 in 2024-04-27T01:23:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:53 smithi162 ceph-mon[37680]: osdmap e609: 8 total, 8 up, 8 in 2024-04-27T01:23:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:53 smithi162 ceph-mon[37680]: osdmap e610: 8 total, 8 up, 8 in 2024-04-27T01:23:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:53 smithi162 ceph-mon[26714]: osdmap e609: 8 total, 8 up, 8 in 2024-04-27T01:23:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:23:54.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:53 smithi162 ceph-mon[26714]: osdmap e610: 8 total, 8 up, 8 in 2024-04-27T01:23:54.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:23:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:54 smithi181 ceph-mon[30237]: pgmap v940: 172 pgs: 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:54 smithi181 ceph-mon[30237]: osdmap e611: 8 total, 8 up, 8 in 2024-04-27T01:23:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:55.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:54 smithi162 ceph-mon[37680]: pgmap v940: 172 pgs: 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:55.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:55.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:54 smithi162 ceph-mon[37680]: osdmap e611: 8 total, 8 up, 8 in 2024-04-27T01:23:55.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:54 smithi162 ceph-mon[26714]: pgmap v940: 172 pgs: 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:23:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:54 smithi162 ceph-mon[26714]: osdmap e611: 8 total, 8 up, 8 in 2024-04-27T01:23:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:23:56.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:56.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:56 smithi162 ceph-mon[37680]: osdmap e612: 8 total, 8 up, 8 in 2024-04-27T01:23:56.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-120"}]: dispatch 2024-04-27T01:23:56.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:56 smithi162 ceph-mon[37680]: pgmap v944: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:56.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:56.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:56 smithi162 ceph-mon[26714]: osdmap e612: 8 total, 8 up, 8 in 2024-04-27T01:23:56.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-120"}]: dispatch 2024-04-27T01:23:56.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:56 smithi162 ceph-mon[26714]: pgmap v944: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:56.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:23:56.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:56 smithi181 ceph-mon[30237]: osdmap e612: 8 total, 8 up, 8 in 2024-04-27T01:23:56.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-120"}]: dispatch 2024-04-27T01:23:56.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:56 smithi181 ceph-mon[30237]: pgmap v944: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:23:57.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-120"}]': finished 2024-04-27T01:23:57.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:57 smithi162 ceph-mon[37680]: osdmap e613: 8 total, 8 up, 8 in 2024-04-27T01:23:57.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-120", "mode": "writeback"}]: dispatch 2024-04-27T01:23:57.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-120"}]': finished 2024-04-27T01:23:57.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:57 smithi162 ceph-mon[26714]: osdmap e613: 8 total, 8 up, 8 in 2024-04-27T01:23:57.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-120", "mode": "writeback"}]: dispatch 2024-04-27T01:23:57.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-120"}]': finished 2024-04-27T01:23:57.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:57 smithi181 ceph-mon[30237]: osdmap e613: 8 total, 8 up, 8 in 2024-04-27T01:23:57.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-120", "mode": "writeback"}]: dispatch 2024-04-27T01:23:58.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:58 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:58.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-120", "mode": "writeback"}]': finished 2024-04-27T01:23:58.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:58 smithi162 ceph-mon[37680]: osdmap e614: 8 total, 8 up, 8 in 2024-04-27T01:23:58.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:58 smithi162 ceph-mon[37680]: pgmap v947: 204 pgs: 204 active+clean; 583 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 256 B/s wr, 1 op/s 2024-04-27T01:23:58.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:58.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:58 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:58.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-120", "mode": "writeback"}]': finished 2024-04-27T01:23:58.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:58 smithi162 ceph-mon[26714]: osdmap e614: 8 total, 8 up, 8 in 2024-04-27T01:23:58.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:58 smithi162 ceph-mon[26714]: pgmap v947: 204 pgs: 204 active+clean; 583 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 256 B/s wr, 1 op/s 2024-04-27T01:23:58.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:58.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:58 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:23:58.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-120", "mode": "writeback"}]': finished 2024-04-27T01:23:58.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:58 smithi181 ceph-mon[30237]: osdmap e614: 8 total, 8 up, 8 in 2024-04-27T01:23:58.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:58 smithi181 ceph-mon[30237]: pgmap v947: 204 pgs: 204 active+clean; 583 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 256 B/s wr, 1 op/s 2024-04-27T01:23:58.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:23:59.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:59.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:59 smithi162 ceph-mon[37680]: osdmap e615: 8 total, 8 up, 8 in 2024-04-27T01:23:59.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:23:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120"}]: dispatch 2024-04-27T01:23:59.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:59.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:59 smithi162 ceph-mon[26714]: osdmap e615: 8 total, 8 up, 8 in 2024-04-27T01:23:59.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:23:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120"}]: dispatch 2024-04-27T01:23:59.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:23:59.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:59 smithi181 ceph-mon[30237]: osdmap e615: 8 total, 8 up, 8 in 2024-04-27T01:23:59.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:23:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120"}]: dispatch 2024-04-27T01:24:00.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:23:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:23:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:24:00.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:00 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:00.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120"}]': finished 2024-04-27T01:24:00.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:00 smithi162 ceph-mon[37680]: osdmap e616: 8 total, 8 up, 8 in 2024-04-27T01:24:00.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:00 smithi162 ceph-mon[37680]: pgmap v950: 204 pgs: 204 active+clean; 583 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:00.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:00 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:00.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120"}]': finished 2024-04-27T01:24:00.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:00 smithi162 ceph-mon[26714]: osdmap e616: 8 total, 8 up, 8 in 2024-04-27T01:24:00.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:00 smithi162 ceph-mon[26714]: pgmap v950: 204 pgs: 204 active+clean; 583 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:00 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-120"}]': finished 2024-04-27T01:24:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:00 smithi181 ceph-mon[30237]: osdmap e616: 8 total, 8 up, 8 in 2024-04-27T01:24:00.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:00 smithi181 ceph-mon[30237]: pgmap v950: 204 pgs: 204 active+clean; 583 KiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:01.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:01 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:01.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:01 smithi162 ceph-mon[37680]: osdmap e617: 8 total, 8 up, 8 in 2024-04-27T01:24:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:01 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:01.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:01 smithi162 ceph-mon[26714]: osdmap e617: 8 total, 8 up, 8 in 2024-04-27T01:24:01.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:01 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:01.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:01 smithi181 ceph-mon[30237]: osdmap e617: 8 total, 8 up, 8 in 2024-04-27T01:24:02.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:02 smithi181 ceph-mon[30237]: pgmap v952: 172 pgs: 172 active+clean; 583 KiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:02.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:02 smithi181 ceph-mon[30237]: osdmap e618: 8 total, 8 up, 8 in 2024-04-27T01:24:02.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:02.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:02 smithi162 ceph-mon[37680]: pgmap v952: 172 pgs: 172 active+clean; 583 KiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:02.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:02 smithi162 ceph-mon[37680]: osdmap e618: 8 total, 8 up, 8 in 2024-04-27T01:24:02.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:02 smithi162 ceph-mon[26714]: pgmap v952: 172 pgs: 172 active+clean; 583 KiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:02 smithi162 ceph-mon[26714]: osdmap e618: 8 total, 8 up, 8 in 2024-04-27T01:24:02.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:03.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:24:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:24:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:03 smithi181 ceph-mon[30237]: osdmap e619: 8 total, 8 up, 8 in 2024-04-27T01:24:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:03.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:03.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:03 smithi162 ceph-mon[26714]: osdmap e619: 8 total, 8 up, 8 in 2024-04-27T01:24:03.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:03.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:03.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:03.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:03 smithi162 ceph-mon[37680]: osdmap e619: 8 total, 8 up, 8 in 2024-04-27T01:24:03.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:03.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:04.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:04 smithi181 ceph-mon[30237]: pgmap v955: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:04.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:04.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:04 smithi181 ceph-mon[30237]: osdmap e620: 8 total, 8 up, 8 in 2024-04-27T01:24:04.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-122"}]: dispatch 2024-04-27T01:24:04.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:04 smithi162 ceph-mon[37680]: pgmap v955: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:04.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:04.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:04 smithi162 ceph-mon[37680]: osdmap e620: 8 total, 8 up, 8 in 2024-04-27T01:24:04.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-122"}]: dispatch 2024-04-27T01:24:04.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:04 smithi162 ceph-mon[26714]: pgmap v955: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:04.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:04.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:04 smithi162 ceph-mon[26714]: osdmap e620: 8 total, 8 up, 8 in 2024-04-27T01:24:04.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-122"}]: dispatch 2024-04-27T01:24:05.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-122"}]': finished 2024-04-27T01:24:05.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:05 smithi181 ceph-mon[30237]: osdmap e621: 8 total, 8 up, 8 in 2024-04-27T01:24:05.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-122", "mode": "writeback"}]: dispatch 2024-04-27T01:24:05.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:05 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:05.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-122", "mode": "writeback"}]': finished 2024-04-27T01:24:05.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-122"}]': finished 2024-04-27T01:24:05.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:05 smithi162 ceph-mon[37680]: osdmap e621: 8 total, 8 up, 8 in 2024-04-27T01:24:05.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-122", "mode": "writeback"}]: dispatch 2024-04-27T01:24:05.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:05 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:05.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-122", "mode": "writeback"}]': finished 2024-04-27T01:24:05.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-122"}]': finished 2024-04-27T01:24:05.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:05 smithi162 ceph-mon[26714]: osdmap e621: 8 total, 8 up, 8 in 2024-04-27T01:24:05.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-122", "mode": "writeback"}]: dispatch 2024-04-27T01:24:05.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:05 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:05.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-122", "mode": "writeback"}]': finished 2024-04-27T01:24:06.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:06 smithi181 ceph-mon[30237]: pgmap v958: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:06.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:06 smithi181 ceph-mon[30237]: osdmap e622: 8 total, 8 up, 8 in 2024-04-27T01:24:06.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:06.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:06 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:06.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:06 smithi162 ceph-mon[37680]: pgmap v958: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:06.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:06 smithi162 ceph-mon[37680]: osdmap e622: 8 total, 8 up, 8 in 2024-04-27T01:24:06.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:06.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:06 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:06.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:06 smithi162 ceph-mon[26714]: pgmap v958: 204 pgs: 32 creating+peering, 172 active+clean; 583 KiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:06.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:06 smithi162 ceph-mon[26714]: osdmap e622: 8 total, 8 up, 8 in 2024-04-27T01:24:06.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:06.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:06 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:07.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:07.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:07 smithi181 ceph-mon[30237]: osdmap e623: 8 total, 8 up, 8 in 2024-04-27T01:24:07.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122"}]: dispatch 2024-04-27T01:24:07.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:07.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:07 smithi162 ceph-mon[37680]: osdmap e623: 8 total, 8 up, 8 in 2024-04-27T01:24:07.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122"}]: dispatch 2024-04-27T01:24:07.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:07.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:07 smithi162 ceph-mon[26714]: osdmap e623: 8 total, 8 up, 8 in 2024-04-27T01:24:07.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122"}]: dispatch 2024-04-27T01:24:09.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:08 smithi181 ceph-mon[30237]: pgmap v961: 204 pgs: 204 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-27T01:24:09.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:08 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:09.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122"}]': finished 2024-04-27T01:24:09.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:08 smithi181 ceph-mon[30237]: osdmap e624: 8 total, 8 up, 8 in 2024-04-27T01:24:09.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:08 smithi162 ceph-mon[37680]: pgmap v961: 204 pgs: 204 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-27T01:24:09.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:08 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:09.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122"}]': finished 2024-04-27T01:24:09.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:08 smithi162 ceph-mon[37680]: osdmap e624: 8 total, 8 up, 8 in 2024-04-27T01:24:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:08 smithi162 ceph-mon[26714]: pgmap v961: 204 pgs: 204 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-27T01:24:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:08 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-122"}]': finished 2024-04-27T01:24:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:08 smithi162 ceph-mon[26714]: osdmap e624: 8 total, 8 up, 8 in 2024-04-27T01:24:10.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:09 smithi181 ceph-mon[30237]: osdmap e625: 8 total, 8 up, 8 in 2024-04-27T01:24:10.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:09 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:10.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:09 smithi181 ceph-mon[30237]: osdmap e626: 8 total, 8 up, 8 in 2024-04-27T01:24:10.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:10.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:24:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:24:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:24:10.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:09 smithi162 ceph-mon[26714]: osdmap e625: 8 total, 8 up, 8 in 2024-04-27T01:24:10.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:09 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:10.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:09 smithi162 ceph-mon[26714]: osdmap e626: 8 total, 8 up, 8 in 2024-04-27T01:24:10.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:10.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:09 smithi162 ceph-mon[37680]: osdmap e625: 8 total, 8 up, 8 in 2024-04-27T01:24:10.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:09 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:10.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:09 smithi162 ceph-mon[37680]: osdmap e626: 8 total, 8 up, 8 in 2024-04-27T01:24:10.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:11.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:10 smithi181 ceph-mon[30237]: pgmap v964: 172 pgs: 172 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:24:11.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:11.009 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:10 smithi181 ceph-mon[30237]: osdmap e627: 8 total, 8 up, 8 in 2024-04-27T01:24:11.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:10 smithi162 ceph-mon[37680]: pgmap v964: 172 pgs: 172 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:24:11.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:11.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:10 smithi162 ceph-mon[37680]: osdmap e627: 8 total, 8 up, 8 in 2024-04-27T01:24:11.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:10 smithi162 ceph-mon[26714]: pgmap v964: 172 pgs: 172 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:24:11.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:11.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:10 smithi162 ceph-mon[26714]: osdmap e627: 8 total, 8 up, 8 in 2024-04-27T01:24:12.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:12 smithi181 ceph-mon[30237]: pgmap v967: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:12.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:12 smithi181 ceph-mon[30237]: osdmap e628: 8 total, 8 up, 8 in 2024-04-27T01:24:12.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:12 smithi162 ceph-mon[37680]: pgmap v967: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:12 smithi162 ceph-mon[37680]: osdmap e628: 8 total, 8 up, 8 in 2024-04-27T01:24:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:12 smithi162 ceph-mon[26714]: pgmap v967: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:12 smithi162 ceph-mon[26714]: osdmap e628: 8 total, 8 up, 8 in 2024-04-27T01:24:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:13.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:24:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:24:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:13 smithi162 ceph-mon[37680]: osdmap e629: 8 total, 8 up, 8 in 2024-04-27T01:24:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-124"}]: dispatch 2024-04-27T01:24:13.737 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:13.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:13.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:13 smithi162 ceph-mon[26714]: osdmap e629: 8 total, 8 up, 8 in 2024-04-27T01:24:13.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-124"}]: dispatch 2024-04-27T01:24:13.737 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:13 smithi181 ceph-mon[30237]: osdmap e629: 8 total, 8 up, 8 in 2024-04-27T01:24:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-124"}]: dispatch 2024-04-27T01:24:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:14.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:14 smithi181 ceph-mon[30237]: pgmap v970: 204 pgs: 204 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:24:14.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-124"}]': finished 2024-04-27T01:24:14.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:14 smithi181 ceph-mon[30237]: osdmap e630: 8 total, 8 up, 8 in 2024-04-27T01:24:14.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-124", "mode": "writeback"}]: dispatch 2024-04-27T01:24:14.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:14 smithi162 ceph-mon[37680]: pgmap v970: 204 pgs: 204 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:24:14.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-124"}]': finished 2024-04-27T01:24:14.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:14 smithi162 ceph-mon[37680]: osdmap e630: 8 total, 8 up, 8 in 2024-04-27T01:24:14.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-124", "mode": "writeback"}]: dispatch 2024-04-27T01:24:14.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:14 smithi162 ceph-mon[26714]: pgmap v970: 204 pgs: 204 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:24:14.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-124"}]': finished 2024-04-27T01:24:14.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:14 smithi162 ceph-mon[26714]: osdmap e630: 8 total, 8 up, 8 in 2024-04-27T01:24:14.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-124", "mode": "writeback"}]: dispatch 2024-04-27T01:24:15.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:15 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:15.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-124", "mode": "writeback"}]': finished 2024-04-27T01:24:15.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:15 smithi181 ceph-mon[30237]: osdmap e631: 8 total, 8 up, 8 in 2024-04-27T01:24:15.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:15 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:15.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-124", "mode": "writeback"}]': finished 2024-04-27T01:24:15.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:15 smithi162 ceph-mon[37680]: osdmap e631: 8 total, 8 up, 8 in 2024-04-27T01:24:15.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:15 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:15.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-124", "mode": "writeback"}]': finished 2024-04-27T01:24:15.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:15 smithi162 ceph-mon[26714]: osdmap e631: 8 total, 8 up, 8 in 2024-04-27T01:24:16.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:16 smithi181 ceph-mon[30237]: pgmap v973: 204 pgs: 204 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:24:16.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:16 smithi181 ceph-mon[30237]: osdmap e632: 8 total, 8 up, 8 in 2024-04-27T01:24:16.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:16.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:16 smithi162 ceph-mon[37680]: pgmap v973: 204 pgs: 204 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:24:16.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:16 smithi162 ceph-mon[37680]: osdmap e632: 8 total, 8 up, 8 in 2024-04-27T01:24:16.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:16 smithi162 ceph-mon[26714]: pgmap v973: 204 pgs: 204 active+clean; 583 KiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-27T01:24:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:16 smithi162 ceph-mon[26714]: osdmap e632: 8 total, 8 up, 8 in 2024-04-27T01:24:16.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:17 smithi181 ceph-mon[30237]: osdmap e633: 8 total, 8 up, 8 in 2024-04-27T01:24:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124"}]: dispatch 2024-04-27T01:24:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:17 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124"}]': finished 2024-04-27T01:24:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:17 smithi181 ceph-mon[30237]: osdmap e634: 8 total, 8 up, 8 in 2024-04-27T01:24:17.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:17.953 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[37680]: osdmap e633: 8 total, 8 up, 8 in 2024-04-27T01:24:17.953 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124"}]: dispatch 2024-04-27T01:24:17.954 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:17.954 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124"}]': finished 2024-04-27T01:24:17.954 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[37680]: osdmap e634: 8 total, 8 up, 8 in 2024-04-27T01:24:17.954 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:17.954 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[26714]: osdmap e633: 8 total, 8 up, 8 in 2024-04-27T01:24:17.954 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124"}]: dispatch 2024-04-27T01:24:17.954 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:17.954 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-124"}]': finished 2024-04-27T01:24:17.954 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:17 smithi162 ceph-mon[26714]: osdmap e634: 8 total, 8 up, 8 in 2024-04-27T01:24:19.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:18 smithi181 ceph-mon[30237]: pgmap v976: 204 pgs: 204 active+clean; 583 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-27T01:24:19.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:18 smithi162 ceph-mon[37680]: pgmap v976: 204 pgs: 204 active+clean; 583 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-27T01:24:19.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:18 smithi162 ceph-mon[26714]: pgmap v976: 204 pgs: 204 active+clean; 583 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-27T01:24:20.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:19 smithi181 ceph-mon[30237]: osdmap e635: 8 total, 8 up, 8 in 2024-04-27T01:24:20.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:19 smithi181 ceph-mon[30237]: osdmap e636: 8 total, 8 up, 8 in 2024-04-27T01:24:20.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:20.085 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:24:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:24:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:24:20.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:19 smithi162 ceph-mon[37680]: osdmap e635: 8 total, 8 up, 8 in 2024-04-27T01:24:20.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:19 smithi162 ceph-mon[37680]: osdmap e636: 8 total, 8 up, 8 in 2024-04-27T01:24:20.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:20.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:19 smithi162 ceph-mon[26714]: osdmap e635: 8 total, 8 up, 8 in 2024-04-27T01:24:20.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:19 smithi162 ceph-mon[26714]: osdmap e636: 8 total, 8 up, 8 in 2024-04-27T01:24:20.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:20 smithi181 ceph-mon[30237]: pgmap v980: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-27T01:24:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:20 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:20 smithi181 ceph-mon[30237]: osdmap e637: 8 total, 8 up, 8 in 2024-04-27T01:24:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:20 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[37680]: pgmap v980: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-27T01:24:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[37680]: osdmap e637: 8 total, 8 up, 8 in 2024-04-27T01:24:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:21.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[26714]: pgmap v980: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-27T01:24:21.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:21.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:21.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[26714]: osdmap e637: 8 total, 8 up, 8 in 2024-04-27T01:24:21.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:21.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:21.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:22.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:22.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:22 smithi162 ceph-mon[26714]: osdmap e638: 8 total, 8 up, 8 in 2024-04-27T01:24:22.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-126"}]: dispatch 2024-04-27T01:24:22.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:22 smithi162 ceph-mon[26714]: pgmap v983: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:22.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:22.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:22 smithi162 ceph-mon[37680]: osdmap e638: 8 total, 8 up, 8 in 2024-04-27T01:24:22.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-126"}]: dispatch 2024-04-27T01:24:22.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:22 smithi162 ceph-mon[37680]: pgmap v983: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:22.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:22 smithi181 ceph-mon[30237]: osdmap e638: 8 total, 8 up, 8 in 2024-04-27T01:24:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-126"}]: dispatch 2024-04-27T01:24:22.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:22 smithi181 ceph-mon[30237]: pgmap v983: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:23.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:24:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:24:23.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-126"}]': finished 2024-04-27T01:24:23.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[37680]: osdmap e639: 8 total, 8 up, 8 in 2024-04-27T01:24:23.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-126", "mode": "writeback"}]: dispatch 2024-04-27T01:24:23.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:24:23.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:23.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:24:23.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:24:23.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-126"}]': finished 2024-04-27T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[26714]: osdmap e639: 8 total, 8 up, 8 in 2024-04-27T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-126", "mode": "writeback"}]: dispatch 2024-04-27T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:24:23.586 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:23 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:23.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-126"}]': finished 2024-04-27T01:24:23.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:23 smithi181 ceph-mon[30237]: osdmap e639: 8 total, 8 up, 8 in 2024-04-27T01:24:23.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-126", "mode": "writeback"}]: dispatch 2024-04-27T01:24:23.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:23 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:24:23.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:23.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:23 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:24:23.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:23 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:24:23.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:23 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:24.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:24 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:24.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-126", "mode": "writeback"}]': finished 2024-04-27T01:24:24.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:24 smithi181 ceph-mon[30237]: osdmap e640: 8 total, 8 up, 8 in 2024-04-27T01:24:24.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:24 smithi181 ceph-mon[30237]: pgmap v986: 204 pgs: 204 active+clean; 583 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:24.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:24.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:24 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-126", "mode": "writeback"}]': finished 2024-04-27T01:24:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:24 smithi162 ceph-mon[37680]: osdmap e640: 8 total, 8 up, 8 in 2024-04-27T01:24:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:24 smithi162 ceph-mon[37680]: pgmap v986: 204 pgs: 204 active+clean; 583 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:24 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-126", "mode": "writeback"}]': finished 2024-04-27T01:24:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:24 smithi162 ceph-mon[26714]: osdmap e640: 8 total, 8 up, 8 in 2024-04-27T01:24:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:24 smithi162 ceph-mon[26714]: pgmap v986: 204 pgs: 204 active+clean; 583 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:25.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:25.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:25 smithi181 ceph-mon[30237]: osdmap e641: 8 total, 8 up, 8 in 2024-04-27T01:24:25.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126"}]: dispatch 2024-04-27T01:24:25.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:25.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:25 smithi162 ceph-mon[37680]: osdmap e641: 8 total, 8 up, 8 in 2024-04-27T01:24:25.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126"}]: dispatch 2024-04-27T01:24:25.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:25.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:25 smithi162 ceph-mon[26714]: osdmap e641: 8 total, 8 up, 8 in 2024-04-27T01:24:25.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126"}]: dispatch 2024-04-27T01:24:26.365 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:24:26.365 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9073 ms) 2024-04-27T01:24:26.365 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-04-27T01:24:26.365 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6041 ms) 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5029 ms) 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26108 ms) 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13345 ms) 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12077 ms) 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-04-27T01:24:26.366 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 00000010 69 73 |is| 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 00000012 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12060 ms) 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: require_osd_release = squid 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6053 ms) 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (560157 ms total) 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1016 ms) 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11042 ms) 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18078 ms) 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-04-27T01:24:26.367 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30138 ms total) 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7065 ms) 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8055 ms) 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: waiting for scrub... 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: done waiting 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24582 ms) 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9900 ms) 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7988 ms) 2024-04-27T01:24:26.368 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-04-27T01:24:26.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8065 ms) 2024-04-27T01:24:26.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-04-27T01:24:26.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10057 ms) 2024-04-27T01:24:26.369 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-04-27T01:24:26.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:26 smithi181 ceph-mon[30237]: pgmap v988: 204 pgs: 204 active+clean; 583 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:26.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:26 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:26.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126"}]': finished 2024-04-27T01:24:26.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:26 smithi181 ceph-mon[30237]: osdmap e642: 8 total, 8 up, 8 in 2024-04-27T01:24:26.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:26 smithi162 ceph-mon[37680]: pgmap v988: 204 pgs: 204 active+clean; 583 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:26.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:26 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:26.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126"}]': finished 2024-04-27T01:24:26.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:26 smithi162 ceph-mon[37680]: osdmap e642: 8 total, 8 up, 8 in 2024-04-27T01:24:26.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:26 smithi162 ceph-mon[26714]: pgmap v988: 204 pgs: 204 active+clean; 583 KiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:26.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:26 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:26.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-126"}]': finished 2024-04-27T01:24:26.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:26 smithi162 ceph-mon[26714]: osdmap e642: 8 total, 8 up, 8 in 2024-04-27T01:24:27.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:27 smithi181 ceph-mon[30237]: osdmap e643: 8 total, 8 up, 8 in 2024-04-27T01:24:27.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:27 smithi162 ceph-mon[37680]: osdmap e643: 8 total, 8 up, 8 in 2024-04-27T01:24:27.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:27 smithi162 ceph-mon[26714]: osdmap e643: 8 total, 8 up, 8 in 2024-04-27T01:24:28.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:28 smithi181 ceph-mon[30237]: pgmap v991: 172 pgs: 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:28.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:28 smithi181 ceph-mon[30237]: osdmap e644: 8 total, 8 up, 8 in 2024-04-27T01:24:28.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:28.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:28 smithi162 ceph-mon[37680]: pgmap v991: 172 pgs: 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:28.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:28 smithi162 ceph-mon[37680]: osdmap e644: 8 total, 8 up, 8 in 2024-04-27T01:24:28.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:28.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:28 smithi162 ceph-mon[26714]: pgmap v991: 172 pgs: 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:28.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:28 smithi162 ceph-mon[26714]: osdmap e644: 8 total, 8 up, 8 in 2024-04-27T01:24:28.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:29.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:29.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:29 smithi162 ceph-mon[37680]: osdmap e645: 8 total, 8 up, 8 in 2024-04-27T01:24:29.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:29.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:29 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:29.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:29.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:29 smithi162 ceph-mon[26714]: osdmap e645: 8 total, 8 up, 8 in 2024-04-27T01:24:29.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:29.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:29 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:29.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:29.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:29 smithi181 ceph-mon[30237]: osdmap e645: 8 total, 8 up, 8 in 2024-04-27T01:24:29.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:29.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:29 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:24:30.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:24:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:24:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:24:30.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:30 smithi181 ceph-mon[30237]: pgmap v994: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:30.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:30.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:30 smithi181 ceph-mon[30237]: osdmap e646: 8 total, 8 up, 8 in 2024-04-27T01:24:30.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-128"}]: dispatch 2024-04-27T01:24:30.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:30 smithi162 ceph-mon[37680]: pgmap v994: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:30.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:30.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:30 smithi162 ceph-mon[37680]: osdmap e646: 8 total, 8 up, 8 in 2024-04-27T01:24:30.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-128"}]: dispatch 2024-04-27T01:24:30.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:30 smithi162 ceph-mon[26714]: pgmap v994: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:30.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:30.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:30 smithi162 ceph-mon[26714]: osdmap e646: 8 total, 8 up, 8 in 2024-04-27T01:24:30.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-128"}]: dispatch 2024-04-27T01:24:31.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-128"}]': finished 2024-04-27T01:24:31.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:31 smithi181 ceph-mon[30237]: osdmap e647: 8 total, 8 up, 8 in 2024-04-27T01:24:31.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-128", "mode": "writeback"}]: dispatch 2024-04-27T01:24:31.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:31 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:31.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:31 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-128", "mode": "writeback"}]': finished 2024-04-27T01:24:31.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:31 smithi181 ceph-mon[30237]: osdmap e648: 8 total, 8 up, 8 in 2024-04-27T01:24:31.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-128"}]': finished 2024-04-27T01:24:31.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[26714]: osdmap e647: 8 total, 8 up, 8 in 2024-04-27T01:24:31.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-128", "mode": "writeback"}]: dispatch 2024-04-27T01:24:31.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:31.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-128", "mode": "writeback"}]': finished 2024-04-27T01:24:31.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[26714]: osdmap e648: 8 total, 8 up, 8 in 2024-04-27T01:24:31.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-128"}]': finished 2024-04-27T01:24:31.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[37680]: osdmap e647: 8 total, 8 up, 8 in 2024-04-27T01:24:31.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-128", "mode": "writeback"}]: dispatch 2024-04-27T01:24:31.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:31.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-128", "mode": "writeback"}]': finished 2024-04-27T01:24:31.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:31 smithi162 ceph-mon[37680]: osdmap e648: 8 total, 8 up, 8 in 2024-04-27T01:24:32.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:32 smithi181 ceph-mon[30237]: pgmap v997: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:32.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:32 smithi162 ceph-mon[26714]: pgmap v997: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:32.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:32 smithi162 ceph-mon[37680]: pgmap v997: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:33.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:24:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:24:33.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:33.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:33.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:34.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:34 smithi181 ceph-mon[30237]: pgmap v999: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 411 B/s wr, 1 op/s 2024-04-27T01:24:34.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:34 smithi162 ceph-mon[37680]: pgmap v999: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 411 B/s wr, 1 op/s 2024-04-27T01:24:34.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:34 smithi162 ceph-mon[26714]: pgmap v999: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 411 B/s wr, 1 op/s 2024-04-27T01:24:36.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:36 smithi181 ceph-mon[30237]: pgmap v1000: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-04-27T01:24:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:36 smithi162 ceph-mon[37680]: pgmap v1000: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-04-27T01:24:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:36.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:36 smithi162 ceph-mon[26714]: pgmap v1000: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-04-27T01:24:36.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:36.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:36.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:37.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:37.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:37 smithi181 ceph-mon[30237]: osdmap e649: 8 total, 8 up, 8 in 2024-04-27T01:24:37.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128"}]: dispatch 2024-04-27T01:24:37.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:37.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:37 smithi162 ceph-mon[37680]: osdmap e649: 8 total, 8 up, 8 in 2024-04-27T01:24:37.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128"}]: dispatch 2024-04-27T01:24:37.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:37.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:37 smithi162 ceph-mon[26714]: osdmap e649: 8 total, 8 up, 8 in 2024-04-27T01:24:37.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128"}]: dispatch 2024-04-27T01:24:39.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:38 smithi181 ceph-mon[30237]: pgmap v1002: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 294 B/s wr, 2 op/s 2024-04-27T01:24:39.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:38 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:39.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128"}]': finished 2024-04-27T01:24:39.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:38 smithi181 ceph-mon[30237]: osdmap e650: 8 total, 8 up, 8 in 2024-04-27T01:24:39.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:38 smithi162 ceph-mon[26714]: pgmap v1002: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 294 B/s wr, 2 op/s 2024-04-27T01:24:39.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:38 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:39.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128"}]': finished 2024-04-27T01:24:39.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:38 smithi162 ceph-mon[26714]: osdmap e650: 8 total, 8 up, 8 in 2024-04-27T01:24:39.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:38 smithi162 ceph-mon[37680]: pgmap v1002: 204 pgs: 204 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 294 B/s wr, 2 op/s 2024-04-27T01:24:39.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:38 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:39.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-128"}]': finished 2024-04-27T01:24:39.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:38 smithi162 ceph-mon[37680]: osdmap e650: 8 total, 8 up, 8 in 2024-04-27T01:24:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:39 smithi181 ceph-mon[30237]: osdmap e651: 8 total, 8 up, 8 in 2024-04-27T01:24:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:39 smithi181 ceph-mon[30237]: osdmap e652: 8 total, 8 up, 8 in 2024-04-27T01:24:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:40.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:24:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:24:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:24:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:39 smithi162 ceph-mon[37680]: osdmap e651: 8 total, 8 up, 8 in 2024-04-27T01:24:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:39 smithi162 ceph-mon[37680]: osdmap e652: 8 total, 8 up, 8 in 2024-04-27T01:24:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:40.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:39 smithi162 ceph-mon[26714]: osdmap e651: 8 total, 8 up, 8 in 2024-04-27T01:24:40.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:39 smithi162 ceph-mon[26714]: osdmap e652: 8 total, 8 up, 8 in 2024-04-27T01:24:40.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:41.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:40 smithi181 ceph-mon[30237]: pgmap v1005: 172 pgs: 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-27T01:24:41.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:41.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:40 smithi181 ceph-mon[30237]: osdmap e653: 8 total, 8 up, 8 in 2024-04-27T01:24:41.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:40 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:41.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:40 smithi162 ceph-mon[37680]: pgmap v1005: 172 pgs: 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-27T01:24:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:40 smithi162 ceph-mon[37680]: osdmap e653: 8 total, 8 up, 8 in 2024-04-27T01:24:41.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:40 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:41.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:40 smithi162 ceph-mon[26714]: pgmap v1005: 172 pgs: 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-27T01:24:41.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:41.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:40 smithi162 ceph-mon[26714]: osdmap e653: 8 total, 8 up, 8 in 2024-04-27T01:24:41.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:40 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:42.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:42 smithi181 ceph-mon[30237]: pgmap v1008: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:42.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:42.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:42 smithi181 ceph-mon[30237]: osdmap e654: 8 total, 8 up, 8 in 2024-04-27T01:24:42.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-130"}]: dispatch 2024-04-27T01:24:42.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:42 smithi162 ceph-mon[37680]: pgmap v1008: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:42.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:42.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:42 smithi162 ceph-mon[37680]: osdmap e654: 8 total, 8 up, 8 in 2024-04-27T01:24:42.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-130"}]: dispatch 2024-04-27T01:24:42.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:42 smithi162 ceph-mon[26714]: pgmap v1008: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 675 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:42.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:42.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:42 smithi162 ceph-mon[26714]: osdmap e654: 8 total, 8 up, 8 in 2024-04-27T01:24:42.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-130"}]: dispatch 2024-04-27T01:24:43.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:24:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:24:43.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-130"}]': finished 2024-04-27T01:24:43.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:43 smithi181 ceph-mon[30237]: osdmap e655: 8 total, 8 up, 8 in 2024-04-27T01:24:43.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-130", "mode": "writeback"}]: dispatch 2024-04-27T01:24:43.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:43.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:43 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:43.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-130", "mode": "writeback"}]': finished 2024-04-27T01:24:43.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-130"}]': finished 2024-04-27T01:24:43.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[26714]: osdmap e655: 8 total, 8 up, 8 in 2024-04-27T01:24:43.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-130", "mode": "writeback"}]: dispatch 2024-04-27T01:24:43.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:43.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:43.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-130", "mode": "writeback"}]': finished 2024-04-27T01:24:43.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-130"}]': finished 2024-04-27T01:24:43.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[37680]: osdmap e655: 8 total, 8 up, 8 in 2024-04-27T01:24:43.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-130", "mode": "writeback"}]: dispatch 2024-04-27T01:24:43.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:43.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:43.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-130", "mode": "writeback"}]': finished 2024-04-27T01:24:44.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:44 smithi181 ceph-mon[30237]: pgmap v1011: 204 pgs: 204 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:44.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:44 smithi181 ceph-mon[30237]: osdmap e656: 8 total, 8 up, 8 in 2024-04-27T01:24:44.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:44.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:44 smithi162 ceph-mon[37680]: pgmap v1011: 204 pgs: 204 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:44.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:44 smithi162 ceph-mon[37680]: osdmap e656: 8 total, 8 up, 8 in 2024-04-27T01:24:44.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:44.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:44 smithi162 ceph-mon[26714]: pgmap v1011: 204 pgs: 204 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:44.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:44 smithi162 ceph-mon[26714]: osdmap e656: 8 total, 8 up, 8 in 2024-04-27T01:24:44.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:45.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:45.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:45 smithi162 ceph-mon[26714]: osdmap e657: 8 total, 8 up, 8 in 2024-04-27T01:24:45.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130"}]: dispatch 2024-04-27T01:24:45.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:45.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:45 smithi162 ceph-mon[37680]: osdmap e657: 8 total, 8 up, 8 in 2024-04-27T01:24:45.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130"}]: dispatch 2024-04-27T01:24:46.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:46.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:45 smithi181 ceph-mon[30237]: osdmap e657: 8 total, 8 up, 8 in 2024-04-27T01:24:46.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130"}]: dispatch 2024-04-27T01:24:46.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:46 smithi162 ceph-mon[37680]: pgmap v1014: 204 pgs: 204 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:46.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:46 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:46.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130"}]': finished 2024-04-27T01:24:46.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:46 smithi162 ceph-mon[37680]: osdmap e658: 8 total, 8 up, 8 in 2024-04-27T01:24:46.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:46 smithi162 ceph-mon[26714]: pgmap v1014: 204 pgs: 204 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:46.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:46 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:46.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130"}]': finished 2024-04-27T01:24:46.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:46 smithi162 ceph-mon[26714]: osdmap e658: 8 total, 8 up, 8 in 2024-04-27T01:24:47.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:46 smithi181 ceph-mon[30237]: pgmap v1014: 204 pgs: 204 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:24:47.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:46 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:47.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-130"}]': finished 2024-04-27T01:24:47.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:46 smithi181 ceph-mon[30237]: osdmap e658: 8 total, 8 up, 8 in 2024-04-27T01:24:47.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:47 smithi162 ceph-mon[37680]: osdmap e659: 8 total, 8 up, 8 in 2024-04-27T01:24:47.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:47 smithi162 ceph-mon[26714]: osdmap e659: 8 total, 8 up, 8 in 2024-04-27T01:24:48.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:47 smithi181 ceph-mon[30237]: osdmap e659: 8 total, 8 up, 8 in 2024-04-27T01:24:48.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:48 smithi162 ceph-mon[37680]: pgmap v1017: 172 pgs: 172 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:48.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:48 smithi162 ceph-mon[37680]: osdmap e660: 8 total, 8 up, 8 in 2024-04-27T01:24:48.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:48.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:48 smithi162 ceph-mon[26714]: pgmap v1017: 172 pgs: 172 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:48.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:48 smithi162 ceph-mon[26714]: osdmap e660: 8 total, 8 up, 8 in 2024-04-27T01:24:48.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:49.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:48 smithi181 ceph-mon[30237]: pgmap v1017: 172 pgs: 172 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:49.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:48 smithi181 ceph-mon[30237]: osdmap e660: 8 total, 8 up, 8 in 2024-04-27T01:24:49.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:24:49.834 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:24:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:24:49.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:49.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[37680]: osdmap e661: 8 total, 8 up, 8 in 2024-04-27T01:24:49.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:49.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:49.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[37680]: osdmap e662: 8 total, 8 up, 8 in 2024-04-27T01:24:49.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]: dispatch 2024-04-27T01:24:49.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:49.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[26714]: osdmap e661: 8 total, 8 up, 8 in 2024-04-27T01:24:49.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:49.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:49.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[26714]: osdmap e662: 8 total, 8 up, 8 in 2024-04-27T01:24:49.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]: dispatch 2024-04-27T01:24:50.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:24:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:49 smithi181 ceph-mon[30237]: osdmap e661: 8 total, 8 up, 8 in 2024-04-27T01:24:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:24:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:24:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:49 smithi181 ceph-mon[30237]: osdmap e662: 8 total, 8 up, 8 in 2024-04-27T01:24:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]: dispatch 2024-04-27T01:24:50.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[37680]: pgmap v1020: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:50.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:50.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:50.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]': finished 2024-04-27T01:24:50.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[37680]: osdmap e663: 8 total, 8 up, 8 in 2024-04-27T01:24:50.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-132", "mode": "writeback"}]: dispatch 2024-04-27T01:24:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[26714]: pgmap v1020: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]': finished 2024-04-27T01:24:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[26714]: osdmap e663: 8 total, 8 up, 8 in 2024-04-27T01:24:50.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-132", "mode": "writeback"}]: dispatch 2024-04-27T01:24:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:50 smithi181 ceph-mon[30237]: pgmap v1020: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:50 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:24:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:24:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]': finished 2024-04-27T01:24:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:50 smithi181 ceph-mon[30237]: osdmap e663: 8 total, 8 up, 8 in 2024-04-27T01:24:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-132", "mode": "writeback"}]: dispatch 2024-04-27T01:24:51.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:51 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:51.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:51 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-132", "mode": "writeback"}]': finished 2024-04-27T01:24:51.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:51 smithi162 ceph-mon[37680]: osdmap e664: 8 total, 8 up, 8 in 2024-04-27T01:24:51.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:51 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:51.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:51 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-132", "mode": "writeback"}]': finished 2024-04-27T01:24:51.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:51 smithi162 ceph-mon[26714]: osdmap e664: 8 total, 8 up, 8 in 2024-04-27T01:24:52.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:51 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:24:52.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:51 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-132", "mode": "writeback"}]': finished 2024-04-27T01:24:52.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:51 smithi181 ceph-mon[30237]: osdmap e664: 8 total, 8 up, 8 in 2024-04-27T01:24:52.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:52 smithi162 ceph-mon[37680]: pgmap v1023: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:52.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:52 smithi162 ceph-mon[37680]: osdmap e665: 8 total, 8 up, 8 in 2024-04-27T01:24:52.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:52 smithi162 ceph-mon[26714]: pgmap v1023: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:52.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:52 smithi162 ceph-mon[26714]: osdmap e665: 8 total, 8 up, 8 in 2024-04-27T01:24:53.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:24:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:24:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:52 smithi181 ceph-mon[30237]: pgmap v1023: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:24:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:52 smithi181 ceph-mon[30237]: osdmap e665: 8 total, 8 up, 8 in 2024-04-27T01:24:53.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:53.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:54.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:24:54.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:54 smithi162 ceph-mon[26714]: pgmap v1026: 204 pgs: 204 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:54.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:54 smithi162 ceph-mon[26714]: osdmap e666: 8 total, 8 up, 8 in 2024-04-27T01:24:54.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:54.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:54 smithi162 ceph-mon[37680]: pgmap v1026: 204 pgs: 204 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:54.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:54 smithi162 ceph-mon[37680]: osdmap e666: 8 total, 8 up, 8 in 2024-04-27T01:24:54.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:54 smithi181 ceph-mon[30237]: pgmap v1026: 204 pgs: 204 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:54 smithi181 ceph-mon[30237]: osdmap e666: 8 total, 8 up, 8 in 2024-04-27T01:24:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:55.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:55.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:55 smithi162 ceph-mon[37680]: osdmap e667: 8 total, 8 up, 8 in 2024-04-27T01:24:55.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]: dispatch 2024-04-27T01:24:55.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:55.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:55 smithi162 ceph-mon[26714]: osdmap e667: 8 total, 8 up, 8 in 2024-04-27T01:24:55.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]: dispatch 2024-04-27T01:24:56.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:56.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:55 smithi181 ceph-mon[30237]: osdmap e667: 8 total, 8 up, 8 in 2024-04-27T01:24:56.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]: dispatch 2024-04-27T01:24:56.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:56 smithi162 ceph-mon[37680]: pgmap v1029: 204 pgs: 204 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:56.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]': finished 2024-04-27T01:24:56.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:56 smithi162 ceph-mon[37680]: osdmap e668: 8 total, 8 up, 8 in 2024-04-27T01:24:56.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:56 smithi162 ceph-mon[26714]: pgmap v1029: 204 pgs: 204 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:56.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]': finished 2024-04-27T01:24:56.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:56 smithi162 ceph-mon[26714]: osdmap e668: 8 total, 8 up, 8 in 2024-04-27T01:24:57.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:56 smithi181 ceph-mon[30237]: pgmap v1029: 204 pgs: 204 active+clean; 583 KiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:24:57.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-132"}]': finished 2024-04-27T01:24:57.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:56 smithi181 ceph-mon[30237]: osdmap e668: 8 total, 8 up, 8 in 2024-04-27T01:24:58.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:57 smithi181 ceph-mon[30237]: osdmap e669: 8 total, 8 up, 8 in 2024-04-27T01:24:58.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:58.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:57 smithi162 ceph-mon[26714]: osdmap e669: 8 total, 8 up, 8 in 2024-04-27T01:24:58.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:58.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:57 smithi162 ceph-mon[37680]: osdmap e669: 8 total, 8 up, 8 in 2024-04-27T01:24:58.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:24:59.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:58 smithi181 ceph-mon[30237]: pgmap v1032: 204 pgs: 204 active+clean; 583 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:24:59.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:59.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:58 smithi181 ceph-mon[30237]: osdmap e670: 8 total, 8 up, 8 in 2024-04-27T01:24:59.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132"}]: dispatch 2024-04-27T01:24:59.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:58 smithi162 ceph-mon[37680]: pgmap v1032: 204 pgs: 204 active+clean; 583 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:24:59.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:59.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:58 smithi162 ceph-mon[37680]: osdmap e670: 8 total, 8 up, 8 in 2024-04-27T01:24:59.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132"}]: dispatch 2024-04-27T01:24:59.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:58 smithi162 ceph-mon[26714]: pgmap v1032: 204 pgs: 204 active+clean; 583 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-27T01:24:59.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:24:59.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:58 smithi162 ceph-mon[26714]: osdmap e670: 8 total, 8 up, 8 in 2024-04-27T01:24:59.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132"}]: dispatch 2024-04-27T01:24:59.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:59 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:24:59.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132"}]': finished 2024-04-27T01:24:59.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:59 smithi181 ceph-mon[30237]: osdmap e671: 8 total, 8 up, 8 in 2024-04-27T01:24:59.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:24:59 smithi181 ceph-mon[30237]: osdmap e672: 8 total, 8 up, 8 in 2024-04-27T01:25:00.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:24:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:24:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:25:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:59 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132"}]': finished 2024-04-27T01:25:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:59 smithi162 ceph-mon[26714]: osdmap e671: 8 total, 8 up, 8 in 2024-04-27T01:25:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:24:59 smithi162 ceph-mon[26714]: osdmap e672: 8 total, 8 up, 8 in 2024-04-27T01:25:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:59 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-132"}]': finished 2024-04-27T01:25:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:59 smithi162 ceph-mon[37680]: osdmap e671: 8 total, 8 up, 8 in 2024-04-27T01:25:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:24:59 smithi162 ceph-mon[37680]: osdmap e672: 8 total, 8 up, 8 in 2024-04-27T01:25:01.010 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:00 smithi181 ceph-mon[30237]: pgmap v1036: 172 pgs: 172 active+clean; 583 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 543 B/s wr, 2 op/s 2024-04-27T01:25:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:00 smithi181 ceph-mon[30237]: osdmap e673: 8 total, 8 up, 8 in 2024-04-27T01:25:01.011 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:01.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:00 smithi162 ceph-mon[26714]: pgmap v1036: 172 pgs: 172 active+clean; 583 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 543 B/s wr, 2 op/s 2024-04-27T01:25:01.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:00 smithi162 ceph-mon[26714]: osdmap e673: 8 total, 8 up, 8 in 2024-04-27T01:25:01.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:01.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:00 smithi162 ceph-mon[37680]: pgmap v1036: 172 pgs: 172 active+clean; 583 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 543 B/s wr, 2 op/s 2024-04-27T01:25:01.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:00 smithi162 ceph-mon[37680]: osdmap e673: 8 total, 8 up, 8 in 2024-04-27T01:25:01.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:02.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:02.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:02 smithi162 ceph-mon[37680]: osdmap e674: 8 total, 8 up, 8 in 2024-04-27T01:25:02.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:02.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:02 smithi162 ceph-mon[37680]: pgmap v1039: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:02.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:02.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:02 smithi162 ceph-mon[26714]: osdmap e674: 8 total, 8 up, 8 in 2024-04-27T01:25:02.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:02.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:02 smithi162 ceph-mon[26714]: pgmap v1039: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:02.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:02.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:02 smithi181 ceph-mon[30237]: osdmap e674: 8 total, 8 up, 8 in 2024-04-27T01:25:02.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:02.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:02 smithi181 ceph-mon[30237]: pgmap v1039: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:03.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:25:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:25:03.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:03.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:03 smithi162 ceph-mon[37680]: osdmap e675: 8 total, 8 up, 8 in 2024-04-27T01:25:03.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-134"}]: dispatch 2024-04-27T01:25:03.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:03.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:03.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:03 smithi162 ceph-mon[26714]: osdmap e675: 8 total, 8 up, 8 in 2024-04-27T01:25:03.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-134"}]: dispatch 2024-04-27T01:25:03.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:03.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:03 smithi181 ceph-mon[30237]: osdmap e675: 8 total, 8 up, 8 in 2024-04-27T01:25:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-134"}]: dispatch 2024-04-27T01:25:03.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:04.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-134"}]': finished 2024-04-27T01:25:04.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:04 smithi162 ceph-mon[37680]: osdmap e676: 8 total, 8 up, 8 in 2024-04-27T01:25:04.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:04 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-134", "mode": "writeback"}]: dispatch 2024-04-27T01:25:04.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:04 smithi162 ceph-mon[37680]: pgmap v1042: 204 pgs: 204 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:04.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:04 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:25:04.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-134"}]': finished 2024-04-27T01:25:04.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:04 smithi162 ceph-mon[26714]: osdmap e676: 8 total, 8 up, 8 in 2024-04-27T01:25:04.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:04 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-134", "mode": "writeback"}]: dispatch 2024-04-27T01:25:04.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:04 smithi162 ceph-mon[26714]: pgmap v1042: 204 pgs: 204 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:04.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:04 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:25:04.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-134"}]': finished 2024-04-27T01:25:04.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:04 smithi181 ceph-mon[30237]: osdmap e676: 8 total, 8 up, 8 in 2024-04-27T01:25:04.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:04 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-134", "mode": "writeback"}]: dispatch 2024-04-27T01:25:04.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:04 smithi181 ceph-mon[30237]: pgmap v1042: 204 pgs: 204 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:04.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:04 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:25:05.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-134", "mode": "writeback"}]': finished 2024-04-27T01:25:05.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:05 smithi162 ceph-mon[37680]: osdmap e677: 8 total, 8 up, 8 in 2024-04-27T01:25:05.585 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:05 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:05.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-134", "mode": "writeback"}]': finished 2024-04-27T01:25:05.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:05 smithi162 ceph-mon[26714]: osdmap e677: 8 total, 8 up, 8 in 2024-04-27T01:25:05.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:05 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:05.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-134", "mode": "writeback"}]': finished 2024-04-27T01:25:05.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:05 smithi181 ceph-mon[30237]: osdmap e677: 8 total, 8 up, 8 in 2024-04-27T01:25:05.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:05 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:06.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:25:06.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:06 smithi181 ceph-mon[30237]: osdmap e678: 8 total, 8 up, 8 in 2024-04-27T01:25:06.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134"}]: dispatch 2024-04-27T01:25:06.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:06 smithi181 ceph-mon[30237]: pgmap v1045: 204 pgs: 204 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:06.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:06 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:06.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:06 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:06.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:25:06.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[37680]: osdmap e678: 8 total, 8 up, 8 in 2024-04-27T01:25:06.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134"}]: dispatch 2024-04-27T01:25:06.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[37680]: pgmap v1045: 204 pgs: 204 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:06.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:06.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:06.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:25:06.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[26714]: osdmap e678: 8 total, 8 up, 8 in 2024-04-27T01:25:06.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134"}]: dispatch 2024-04-27T01:25:06.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[26714]: pgmap v1045: 204 pgs: 204 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:06.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:06.836 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:06 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:08.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:07 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:08.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134"}]': finished 2024-04-27T01:25:08.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:07 smithi181 ceph-mon[30237]: osdmap e679: 8 total, 8 up, 8 in 2024-04-27T01:25:08.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:07 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:08.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134"}]': finished 2024-04-27T01:25:08.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:07 smithi162 ceph-mon[37680]: osdmap e679: 8 total, 8 up, 8 in 2024-04-27T01:25:08.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:07 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:08.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-134"}]': finished 2024-04-27T01:25:08.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:07 smithi162 ceph-mon[26714]: osdmap e679: 8 total, 8 up, 8 in 2024-04-27T01:25:09.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:08 smithi181 ceph-mon[30237]: osdmap e680: 8 total, 8 up, 8 in 2024-04-27T01:25:09.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:08 smithi181 ceph-mon[30237]: pgmap v1048: 172 pgs: 172 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:25:09.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:08 smithi181 ceph-mon[30237]: osdmap e681: 8 total, 8 up, 8 in 2024-04-27T01:25:09.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:09.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:08 smithi162 ceph-mon[37680]: osdmap e680: 8 total, 8 up, 8 in 2024-04-27T01:25:09.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:08 smithi162 ceph-mon[37680]: pgmap v1048: 172 pgs: 172 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:25:09.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:08 smithi162 ceph-mon[37680]: osdmap e681: 8 total, 8 up, 8 in 2024-04-27T01:25:09.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:08 smithi162 ceph-mon[26714]: osdmap e680: 8 total, 8 up, 8 in 2024-04-27T01:25:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:08 smithi162 ceph-mon[26714]: pgmap v1048: 172 pgs: 172 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:25:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:08 smithi162 ceph-mon[26714]: osdmap e681: 8 total, 8 up, 8 in 2024-04-27T01:25:09.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:10.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:25:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:25:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:25:10.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:10.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:10 smithi181 ceph-mon[30237]: osdmap e682: 8 total, 8 up, 8 in 2024-04-27T01:25:10.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:10 smithi181 ceph-mon[30237]: pgmap v1051: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:25:10.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:10.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:10.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:10 smithi162 ceph-mon[37680]: osdmap e682: 8 total, 8 up, 8 in 2024-04-27T01:25:10.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:10 smithi162 ceph-mon[37680]: pgmap v1051: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:25:10.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:10.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:10.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:10 smithi162 ceph-mon[26714]: osdmap e682: 8 total, 8 up, 8 in 2024-04-27T01:25:10.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:10 smithi162 ceph-mon[26714]: pgmap v1051: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-27T01:25:10.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:11.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:11.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:11 smithi181 ceph-mon[30237]: osdmap e683: 8 total, 8 up, 8 in 2024-04-27T01:25:11.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:11 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-136"}]: dispatch 2024-04-27T01:25:11.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:11.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:11 smithi162 ceph-mon[37680]: osdmap e683: 8 total, 8 up, 8 in 2024-04-27T01:25:11.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:11 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-136"}]: dispatch 2024-04-27T01:25:11.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:11.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:11 smithi162 ceph-mon[26714]: osdmap e683: 8 total, 8 up, 8 in 2024-04-27T01:25:11.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:11 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-136"}]: dispatch 2024-04-27T01:25:12.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:12 smithi181 ceph-mon[30237]: pgmap v1053: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:12.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-136"}]': finished 2024-04-27T01:25:12.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:12 smithi181 ceph-mon[30237]: osdmap e684: 8 total, 8 up, 8 in 2024-04-27T01:25:12.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-136", "mode": "writeback"}]: dispatch 2024-04-27T01:25:12.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:12 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:25:12.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-136", "mode": "writeback"}]': finished 2024-04-27T01:25:12.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[37680]: pgmap v1053: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-136"}]': finished 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[37680]: osdmap e684: 8 total, 8 up, 8 in 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-136", "mode": "writeback"}]: dispatch 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-136", "mode": "writeback"}]': finished 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[26714]: pgmap v1053: 204 pgs: 32 unknown, 172 active+clean; 583 KiB data, 683 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-136"}]': finished 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[26714]: osdmap e684: 8 total, 8 up, 8 in 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-136", "mode": "writeback"}]: dispatch 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:25:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-136", "mode": "writeback"}]': finished 2024-04-27T01:25:13.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:25:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:25:13.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:13 smithi162 ceph-mon[37680]: osdmap e685: 8 total, 8 up, 8 in 2024-04-27T01:25:13.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:13.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:13.735 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:13 smithi162 ceph-mon[26714]: osdmap e685: 8 total, 8 up, 8 in 2024-04-27T01:25:13.735 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:13.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:13.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:13 smithi181 ceph-mon[30237]: osdmap e685: 8 total, 8 up, 8 in 2024-04-27T01:25:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:13.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:14.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:14 smithi181 ceph-mon[30237]: pgmap v1056: 204 pgs: 204 active+clean; 583 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:14.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:25:14.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:14 smithi181 ceph-mon[30237]: osdmap e686: 8 total, 8 up, 8 in 2024-04-27T01:25:14.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136"}]: dispatch 2024-04-27T01:25:14.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:14 smithi162 ceph-mon[37680]: pgmap v1056: 204 pgs: 204 active+clean; 583 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:14.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:25:14.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:14 smithi162 ceph-mon[37680]: osdmap e686: 8 total, 8 up, 8 in 2024-04-27T01:25:14.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136"}]: dispatch 2024-04-27T01:25:14.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:14 smithi162 ceph-mon[26714]: pgmap v1056: 204 pgs: 204 active+clean; 583 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:14.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:25:14.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:14 smithi162 ceph-mon[26714]: osdmap e686: 8 total, 8 up, 8 in 2024-04-27T01:25:14.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136"}]: dispatch 2024-04-27T01:25:15.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:15 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:15.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136"}]': finished 2024-04-27T01:25:15.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:15 smithi181 ceph-mon[30237]: osdmap e687: 8 total, 8 up, 8 in 2024-04-27T01:25:15.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:15 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:15.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136"}]': finished 2024-04-27T01:25:15.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:15 smithi162 ceph-mon[26714]: osdmap e687: 8 total, 8 up, 8 in 2024-04-27T01:25:15.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:15 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:15.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-136"}]': finished 2024-04-27T01:25:15.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:15 smithi162 ceph-mon[37680]: osdmap e687: 8 total, 8 up, 8 in 2024-04-27T01:25:16.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:16 smithi181 ceph-mon[30237]: pgmap v1059: 204 pgs: 204 active+clean; 583 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:16.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:16 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:25:16.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:16 smithi181 ceph-mon[30237]: osdmap e688: 8 total, 8 up, 8 in 2024-04-27T01:25:16.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:16 smithi162 ceph-mon[26714]: pgmap v1059: 204 pgs: 204 active+clean; 583 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:16.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:16 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:25:16.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:16 smithi162 ceph-mon[26714]: osdmap e688: 8 total, 8 up, 8 in 2024-04-27T01:25:16.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:16 smithi162 ceph-mon[37680]: pgmap v1059: 204 pgs: 204 active+clean; 583 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:16.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:16 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:25:16.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:16 smithi162 ceph-mon[37680]: osdmap e688: 8 total, 8 up, 8 in 2024-04-27T01:25:17.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:17 smithi181 ceph-mon[30237]: osdmap e689: 8 total, 8 up, 8 in 2024-04-27T01:25:17.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:17.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:17 smithi162 ceph-mon[26714]: osdmap e689: 8 total, 8 up, 8 in 2024-04-27T01:25:17.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:17.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:17 smithi162 ceph-mon[37680]: osdmap e689: 8 total, 8 up, 8 in 2024-04-27T01:25:17.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:18.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:18 smithi181 ceph-mon[30237]: pgmap v1062: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-27T01:25:18.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:18.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:18 smithi181 ceph-mon[30237]: osdmap e690: 8 total, 8 up, 8 in 2024-04-27T01:25:18.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:18.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:18 smithi162 ceph-mon[26714]: pgmap v1062: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-27T01:25:18.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:18.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:18 smithi162 ceph-mon[26714]: osdmap e690: 8 total, 8 up, 8 in 2024-04-27T01:25:18.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:18.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:18 smithi162 ceph-mon[37680]: pgmap v1062: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-27T01:25:18.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:18.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:18 smithi162 ceph-mon[37680]: osdmap e690: 8 total, 8 up, 8 in 2024-04-27T01:25:18.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:19.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:19.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[37680]: osdmap e691: 8 total, 8 up, 8 in 2024-04-27T01:25:19.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:25:19.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:25:19.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[37680]: osdmap e692: 8 total, 8 up, 8 in 2024-04-27T01:25:19.727 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:25:19.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:19.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[26714]: osdmap e691: 8 total, 8 up, 8 in 2024-04-27T01:25:19.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:25:19.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:25:19.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[26714]: osdmap e692: 8 total, 8 up, 8 in 2024-04-27T01:25:19.727 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:25:19.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:19.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:19 smithi181 ceph-mon[30237]: osdmap e691: 8 total, 8 up, 8 in 2024-04-27T01:25:19.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:25:19.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:25:19.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:19 smithi181 ceph-mon[30237]: osdmap e692: 8 total, 8 up, 8 in 2024-04-27T01:25:19.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:25:20.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:25:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:25:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:25:20.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:20 smithi181 ceph-mon[30237]: pgmap v1065: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-27T01:25:20.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:25:20.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:20 smithi181 ceph-mon[30237]: osdmap e693: 8 total, 8 up, 8 in 2024-04-27T01:25:20.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:25:20.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:20 smithi162 ceph-mon[37680]: pgmap v1065: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-27T01:25:20.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:25:20.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:20 smithi162 ceph-mon[37680]: osdmap e693: 8 total, 8 up, 8 in 2024-04-27T01:25:20.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:25:20.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:20 smithi162 ceph-mon[26714]: pgmap v1065: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-27T01:25:20.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:25:20.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:20 smithi162 ceph-mon[26714]: osdmap e693: 8 total, 8 up, 8 in 2024-04-27T01:25:20.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-27T01:25:21.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:21 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:21.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:21 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:21.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:25:21.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:21 smithi162 ceph-mon[37680]: osdmap e694: 8 total, 8 up, 8 in 2024-04-27T01:25:21.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:21 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:21.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:21 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:21.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:25:21.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:21 smithi162 ceph-mon[26714]: osdmap e694: 8 total, 8 up, 8 in 2024-04-27T01:25:22.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:21 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:22.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:21 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:22.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-27T01:25:22.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:21 smithi181 ceph-mon[30237]: osdmap e694: 8 total, 8 up, 8 in 2024-04-27T01:25:22.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:22 smithi181 ceph-mon[30237]: pgmap v1068: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:22.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:22.823 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138"}]: dispatch 2024-04-27T01:25:22.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:22 smithi162 ceph-mon[37680]: pgmap v1068: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:22.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:22.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138"}]: dispatch 2024-04-27T01:25:22.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:22 smithi162 ceph-mon[26714]: pgmap v1068: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:22.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:22.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138"}]: dispatch 2024-04-27T01:25:23.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:25:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:25:23.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138"}]': finished 2024-04-27T01:25:23.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:23 smithi162 ceph-mon[26714]: osdmap e695: 8 total, 8 up, 8 in 2024-04-27T01:25:23.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:23.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:23 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:25:23.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138"}]': finished 2024-04-27T01:25:23.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:23 smithi162 ceph-mon[37680]: osdmap e695: 8 total, 8 up, 8 in 2024-04-27T01:25:23.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:23.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:23 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:25:24.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-138"}]': finished 2024-04-27T01:25:24.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:23 smithi181 ceph-mon[30237]: osdmap e695: 8 total, 8 up, 8 in 2024-04-27T01:25:24.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:24.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:23 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:25:24.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[37680]: pgmap v1071: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:24.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:25:24.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[37680]: osdmap e696: 8 total, 8 up, 8 in 2024-04-27T01:25:24.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[26714]: pgmap v1071: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[26714]: osdmap e696: 8 total, 8 up, 8 in 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:25:24.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:24 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:24 smithi181 ceph-mon[30237]: pgmap v1071: 204 pgs: 204 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:24 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:25:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:24 smithi181 ceph-mon[30237]: osdmap e696: 8 total, 8 up, 8 in 2024-04-27T01:25:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:24 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:24 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:25.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:24 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:25:25.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:24 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:25:25.008 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:24 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:25.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:25 smithi162 ceph-mon[37680]: osdmap e697: 8 total, 8 up, 8 in 2024-04-27T01:25:25.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:25.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:25.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:25 smithi162 ceph-mon[37680]: osdmap e698: 8 total, 8 up, 8 in 2024-04-27T01:25:25.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:25 smithi162 ceph-mon[26714]: osdmap e697: 8 total, 8 up, 8 in 2024-04-27T01:25:25.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:25.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:25.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:25 smithi162 ceph-mon[26714]: osdmap e698: 8 total, 8 up, 8 in 2024-04-27T01:25:26.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:25 smithi181 ceph-mon[30237]: osdmap e697: 8 total, 8 up, 8 in 2024-04-27T01:25:26.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:26.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:26.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:25 smithi181 ceph-mon[30237]: osdmap e698: 8 total, 8 up, 8 in 2024-04-27T01:25:26.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:26 smithi162 ceph-mon[37680]: pgmap v1074: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:26.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:26.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:26.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:26 smithi162 ceph-mon[37680]: osdmap e699: 8 total, 8 up, 8 in 2024-04-27T01:25:26.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-27T01:25:26.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:26 smithi162 ceph-mon[26714]: pgmap v1074: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:26.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:26.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:26.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:26 smithi162 ceph-mon[26714]: osdmap e699: 8 total, 8 up, 8 in 2024-04-27T01:25:26.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-27T01:25:27.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:26 smithi181 ceph-mon[30237]: pgmap v1074: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:27.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:27.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:27.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:26 smithi181 ceph-mon[30237]: osdmap e699: 8 total, 8 up, 8 in 2024-04-27T01:25:27.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-27T01:25:28.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:28 smithi162 ceph-mon[37680]: pgmap v1077: 204 pgs: 204 active+clean; 4.4 MiB data, 713 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:28.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_count","val": "3"}]': finished 2024-04-27T01:25:28.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:28 smithi162 ceph-mon[37680]: osdmap e700: 8 total, 8 up, 8 in 2024-04-27T01:25:28.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:28 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-27T01:25:28.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:28 smithi162 ceph-mon[26714]: pgmap v1077: 204 pgs: 204 active+clean; 4.4 MiB data, 713 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:28.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_count","val": "3"}]': finished 2024-04-27T01:25:28.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:28 smithi162 ceph-mon[26714]: osdmap e700: 8 total, 8 up, 8 in 2024-04-27T01:25:28.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:28 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-27T01:25:29.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:28 smithi181 ceph-mon[30237]: pgmap v1077: 204 pgs: 204 active+clean; 4.4 MiB data, 713 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:29.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_count","val": "3"}]': finished 2024-04-27T01:25:29.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:28 smithi181 ceph-mon[30237]: osdmap e700: 8 total, 8 up, 8 in 2024-04-27T01:25:29.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:28 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-27T01:25:29.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_period","val": "3"}]': finished 2024-04-27T01:25:29.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[37680]: osdmap e701: 8 total, 8 up, 8 in 2024-04-27T01:25:29.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:25:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:25:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:25:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[37680]: osdmap e702: 8 total, 8 up, 8 in 2024-04-27T01:25:29.836 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-27T01:25:29.837 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:25:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:25:29.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_period","val": "3"}]': finished 2024-04-27T01:25:29.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[26714]: osdmap e701: 8 total, 8 up, 8 in 2024-04-27T01:25:29.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:25:29.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:25:29.838 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:25:29.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[26714]: osdmap e702: 8 total, 8 up, 8 in 2024-04-27T01:25:29.839 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:29 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-27T01:25:30.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_period","val": "3"}]': finished 2024-04-27T01:25:30.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:29 smithi181 ceph-mon[30237]: osdmap e701: 8 total, 8 up, 8 in 2024-04-27T01:25:30.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:25:30.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:29 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:25:30.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:25:30.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:29 smithi181 ceph-mon[30237]: osdmap e702: 8 total, 8 up, 8 in 2024-04-27T01:25:30.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:29 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-27T01:25:30.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:30 smithi162 ceph-mon[37680]: pgmap v1080: 204 pgs: 204 active+clean; 4.4 MiB data, 713 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:30.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:30 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-27T01:25:30.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:30 smithi162 ceph-mon[37680]: osdmap e703: 8 total, 8 up, 8 in 2024-04-27T01:25:30.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:30 smithi162 ceph-mon[26714]: pgmap v1080: 204 pgs: 204 active+clean; 4.4 MiB data, 713 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:30.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:30 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-27T01:25:30.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:30 smithi162 ceph-mon[26714]: osdmap e703: 8 total, 8 up, 8 in 2024-04-27T01:25:31.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:30 smithi181 ceph-mon[30237]: pgmap v1080: 204 pgs: 204 active+clean; 4.4 MiB data, 713 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:31.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:30 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-27T01:25:31.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:30 smithi181 ceph-mon[30237]: osdmap e703: 8 total, 8 up, 8 in 2024-04-27T01:25:32.828 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:32 smithi181 ceph-mon[30237]: pgmap v1083: 204 pgs: 204 active+clean; 4.4 MiB data, 713 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:32.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:32 smithi162 ceph-mon[26714]: pgmap v1083: 204 pgs: 204 active+clean; 4.4 MiB data, 713 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:32.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:32 smithi162 ceph-mon[37680]: pgmap v1083: 204 pgs: 204 active+clean; 4.4 MiB data, 713 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:33.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:25:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:25:33.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:33.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:34.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:34.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:34 smithi162 ceph-mon[37680]: pgmap v1084: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 882 B/s rd, 2.8 KiB/s wr, 1 op/s 2024-04-27T01:25:34.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:34 smithi162 ceph-mon[26714]: pgmap v1084: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 882 B/s rd, 2.8 KiB/s wr, 1 op/s 2024-04-27T01:25:35.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:34 smithi181 ceph-mon[30237]: pgmap v1084: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 882 B/s rd, 2.8 KiB/s wr, 1 op/s 2024-04-27T01:25:36.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:35 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:36.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:35 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:36.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:35 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:37.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:36 smithi181 ceph-mon[30237]: pgmap v1085: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-04-27T01:25:37.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:36 smithi162 ceph-mon[37680]: pgmap v1085: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-04-27T01:25:37.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:36 smithi162 ceph-mon[26714]: pgmap v1085: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 752 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-04-27T01:25:39.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:38 smithi181 ceph-mon[30237]: pgmap v1086: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-04-27T01:25:39.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:38 smithi162 ceph-mon[26714]: pgmap v1086: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-04-27T01:25:39.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:38 smithi162 ceph-mon[37680]: pgmap v1086: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-04-27T01:25:40.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:25:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:25:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:25:41.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:40 smithi181 ceph-mon[30237]: pgmap v1087: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 5.9 KiB/s wr, 1 op/s 2024-04-27T01:25:41.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:40 smithi162 ceph-mon[37680]: pgmap v1087: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 5.9 KiB/s wr, 1 op/s 2024-04-27T01:25:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:40 smithi162 ceph-mon[26714]: pgmap v1087: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 5.9 KiB/s wr, 1 op/s 2024-04-27T01:25:43.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:25:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:25:43.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:42 smithi181 ceph-mon[30237]: pgmap v1088: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 5.4 KiB/s wr, 1 op/s 2024-04-27T01:25:43.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:42 smithi162 ceph-mon[37680]: pgmap v1088: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 5.4 KiB/s wr, 1 op/s 2024-04-27T01:25:43.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:42 smithi162 ceph-mon[26714]: pgmap v1088: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 5.4 KiB/s wr, 1 op/s 2024-04-27T01:25:43.986 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:43.986 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140"}]: dispatch 2024-04-27T01:25:43.986 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:43.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:43.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140"}]: dispatch 2024-04-27T01:25:43.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:44.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:44.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140"}]: dispatch 2024-04-27T01:25:44.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:45.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:44 smithi181 ceph-mon[30237]: pgmap v1089: 204 pgs: 204 active+clean; 4.5 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-04-27T01:25:45.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140"}]': finished 2024-04-27T01:25:45.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:44 smithi181 ceph-mon[30237]: osdmap e704: 8 total, 8 up, 8 in 2024-04-27T01:25:45.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:44 smithi162 ceph-mon[37680]: pgmap v1089: 204 pgs: 204 active+clean; 4.5 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-04-27T01:25:45.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140"}]': finished 2024-04-27T01:25:45.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:44 smithi162 ceph-mon[37680]: osdmap e704: 8 total, 8 up, 8 in 2024-04-27T01:25:45.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:44 smithi162 ceph-mon[26714]: pgmap v1089: 204 pgs: 204 active+clean; 4.5 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-04-27T01:25:45.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-140"}]': finished 2024-04-27T01:25:45.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:44 smithi162 ceph-mon[26714]: osdmap e704: 8 total, 8 up, 8 in 2024-04-27T01:25:46.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:45 smithi181 ceph-mon[30237]: osdmap e705: 8 total, 8 up, 8 in 2024-04-27T01:25:46.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:45 smithi162 ceph-mon[37680]: osdmap e705: 8 total, 8 up, 8 in 2024-04-27T01:25:46.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:45 smithi162 ceph-mon[26714]: osdmap e705: 8 total, 8 up, 8 in 2024-04-27T01:25:47.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:46 smithi181 ceph-mon[30237]: pgmap v1092: 172 pgs: 172 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-27T01:25:47.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:46 smithi181 ceph-mon[30237]: osdmap e706: 8 total, 8 up, 8 in 2024-04-27T01:25:47.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:47.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:46 smithi162 ceph-mon[37680]: pgmap v1092: 172 pgs: 172 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-27T01:25:47.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:46 smithi162 ceph-mon[37680]: osdmap e706: 8 total, 8 up, 8 in 2024-04-27T01:25:47.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:47.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:46 smithi162 ceph-mon[26714]: pgmap v1092: 172 pgs: 172 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-27T01:25:47.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:46 smithi162 ceph-mon[26714]: osdmap e706: 8 total, 8 up, 8 in 2024-04-27T01:25:47.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:25:48.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:48.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:47 smithi181 ceph-mon[30237]: osdmap e707: 8 total, 8 up, 8 in 2024-04-27T01:25:48.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:47 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:48.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:48.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:47 smithi162 ceph-mon[37680]: osdmap e707: 8 total, 8 up, 8 in 2024-04-27T01:25:48.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:47 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:48.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:25:48.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:47 smithi162 ceph-mon[26714]: osdmap e707: 8 total, 8 up, 8 in 2024-04-27T01:25:48.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:47 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:25:49.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:48 smithi181 ceph-mon[30237]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:25:49.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:49.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:48 smithi181 ceph-mon[30237]: osdmap e708: 8 total, 8 up, 8 in 2024-04-27T01:25:49.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-142"}]: dispatch 2024-04-27T01:25:49.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:48 smithi162 ceph-mon[37680]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:25:49.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:49.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:48 smithi162 ceph-mon[37680]: osdmap e708: 8 total, 8 up, 8 in 2024-04-27T01:25:49.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-142"}]: dispatch 2024-04-27T01:25:49.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:48 smithi162 ceph-mon[26714]: pgmap v1095: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-27T01:25:49.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:25:49.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:48 smithi162 ceph-mon[26714]: osdmap e708: 8 total, 8 up, 8 in 2024-04-27T01:25:49.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-142"}]: dispatch 2024-04-27T01:25:50.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-142"}]': finished 2024-04-27T01:25:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:49 smithi181 ceph-mon[30237]: osdmap e709: 8 total, 8 up, 8 in 2024-04-27T01:25:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-142", "mode": "writeback"}]: dispatch 2024-04-27T01:25:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:49 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:25:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-142", "mode": "writeback"}]': finished 2024-04-27T01:25:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:49 smithi181 ceph-mon[30237]: osdmap e710: 8 total, 8 up, 8 in 2024-04-27T01:25:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:25:50.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:25:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:25:50.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-142"}]': finished 2024-04-27T01:25:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[37680]: osdmap e709: 8 total, 8 up, 8 in 2024-04-27T01:25:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-142", "mode": "writeback"}]: dispatch 2024-04-27T01:25:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:25:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-142", "mode": "writeback"}]': finished 2024-04-27T01:25:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[37680]: osdmap e710: 8 total, 8 up, 8 in 2024-04-27T01:25:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:25:50.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-142"}]': finished 2024-04-27T01:25:50.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[26714]: osdmap e709: 8 total, 8 up, 8 in 2024-04-27T01:25:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-142", "mode": "writeback"}]: dispatch 2024-04-27T01:25:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:25:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-142", "mode": "writeback"}]': finished 2024-04-27T01:25:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[26714]: osdmap e710: 8 total, 8 up, 8 in 2024-04-27T01:25:50.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:25:51.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:50 smithi181 ceph-mon[30237]: pgmap v1098: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-27T01:25:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:50 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:25:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:50 smithi181 ceph-mon[30237]: osdmap e711: 8 total, 8 up, 8 in 2024-04-27T01:25:51.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:25:51.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[37680]: pgmap v1098: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-27T01:25:51.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:51.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:51.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:25:51.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[37680]: osdmap e711: 8 total, 8 up, 8 in 2024-04-27T01:25:51.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:25:51.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[26714]: pgmap v1098: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-27T01:25:51.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:25:51.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:25:51.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:25:51.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[26714]: osdmap e711: 8 total, 8 up, 8 in 2024-04-27T01:25:51.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:25:53.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:25:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:25:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:52 smithi181 ceph-mon[30237]: pgmap v1101: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:25:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:52 smithi181 ceph-mon[30237]: osdmap e712: 8 total, 8 up, 8 in 2024-04-27T01:25:53.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:25:53.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:52 smithi162 ceph-mon[37680]: pgmap v1101: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:25:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:52 smithi162 ceph-mon[37680]: osdmap e712: 8 total, 8 up, 8 in 2024-04-27T01:25:53.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:25:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:52 smithi162 ceph-mon[26714]: pgmap v1101: 204 pgs: 204 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:25:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:25:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:52 smithi162 ceph-mon[26714]: osdmap e712: 8 total, 8 up, 8 in 2024-04-27T01:25:53.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:25:54.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:53 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:54.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:25:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:53 smithi181 ceph-mon[30237]: osdmap e713: 8 total, 8 up, 8 in 2024-04-27T01:25:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:25:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:25:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:53 smithi181 ceph-mon[30237]: osdmap e714: 8 total, 8 up, 8 in 2024-04-27T01:25:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-27T01:25:54.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:54.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:25:54.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[37680]: osdmap e713: 8 total, 8 up, 8 in 2024-04-27T01:25:54.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[37680]: osdmap e714: 8 total, 8 up, 8 in 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[26714]: osdmap e713: 8 total, 8 up, 8 in 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[26714]: osdmap e714: 8 total, 8 up, 8 in 2024-04-27T01:25:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-27T01:25:55.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:54 smithi181 ceph-mon[30237]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:55.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-27T01:25:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:54 smithi181 ceph-mon[30237]: osdmap e715: 8 total, 8 up, 8 in 2024-04-27T01:25:55.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:54 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-27T01:25:55.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:54 smithi162 ceph-mon[37680]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:55.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-27T01:25:55.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:54 smithi162 ceph-mon[37680]: osdmap e715: 8 total, 8 up, 8 in 2024-04-27T01:25:55.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:54 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-27T01:25:55.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:54 smithi162 ceph-mon[26714]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-27T01:25:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:54 smithi162 ceph-mon[26714]: osdmap e715: 8 total, 8 up, 8 in 2024-04-27T01:25:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:54 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-27T01:25:57.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:56 smithi181 ceph-mon[30237]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:57.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-27T01:25:57.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:56 smithi181 ceph-mon[30237]: osdmap e716: 8 total, 8 up, 8 in 2024-04-27T01:25:57.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:57.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:56 smithi162 ceph-mon[37680]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:57.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-27T01:25:57.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:56 smithi162 ceph-mon[37680]: osdmap e716: 8 total, 8 up, 8 in 2024-04-27T01:25:57.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:57.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:56 smithi162 ceph-mon[26714]: pgmap v1107: 204 pgs: 204 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:25:57.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-27T01:25:57.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:56 smithi162 ceph-mon[26714]: osdmap e716: 8 total, 8 up, 8 in 2024-04-27T01:25:57.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:58.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:25:58.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:57 smithi181 ceph-mon[30237]: osdmap e717: 8 total, 8 up, 8 in 2024-04-27T01:25:58.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142"}]: dispatch 2024-04-27T01:25:58.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:25:58.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:57 smithi162 ceph-mon[37680]: osdmap e717: 8 total, 8 up, 8 in 2024-04-27T01:25:58.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142"}]: dispatch 2024-04-27T01:25:58.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:25:58.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:57 smithi162 ceph-mon[26714]: osdmap e717: 8 total, 8 up, 8 in 2024-04-27T01:25:58.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142"}]: dispatch 2024-04-27T01:25:59.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:58 smithi181 ceph-mon[30237]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:25:59.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142"}]': finished 2024-04-27T01:25:59.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:58 smithi181 ceph-mon[30237]: osdmap e718: 8 total, 8 up, 8 in 2024-04-27T01:25:59.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:59.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142"}]: dispatch 2024-04-27T01:25:59.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:58 smithi162 ceph-mon[37680]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:25:59.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142"}]': finished 2024-04-27T01:25:59.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:58 smithi162 ceph-mon[37680]: osdmap e718: 8 total, 8 up, 8 in 2024-04-27T01:25:59.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:59.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142"}]: dispatch 2024-04-27T01:25:59.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:58 smithi162 ceph-mon[26714]: pgmap v1110: 204 pgs: 204 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:25:59.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142"}]': finished 2024-04-27T01:25:59.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:58 smithi162 ceph-mon[26714]: osdmap e718: 8 total, 8 up, 8 in 2024-04-27T01:25:59.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:25:59.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-142"}]: dispatch 2024-04-27T01:26:00.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:25:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:25:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:26:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:59 smithi162 ceph-mon[37680]: osdmap e719: 8 total, 8 up, 8 in 2024-04-27T01:26:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:59 smithi162 ceph-mon[37680]: osdmap e720: 8 total, 8 up, 8 in 2024-04-27T01:26:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:25:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:59 smithi162 ceph-mon[26714]: osdmap e719: 8 total, 8 up, 8 in 2024-04-27T01:26:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:59 smithi162 ceph-mon[26714]: osdmap e720: 8 total, 8 up, 8 in 2024-04-27T01:26:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:25:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:00.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:59 smithi181 ceph-mon[30237]: osdmap e719: 8 total, 8 up, 8 in 2024-04-27T01:26:00.328 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:59 smithi181 ceph-mon[30237]: osdmap e720: 8 total, 8 up, 8 in 2024-04-27T01:26:00.328 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:25:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:01.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:00 smithi181 ceph-mon[30237]: pgmap v1113: 172 pgs: 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:26:01.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:01.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:00 smithi181 ceph-mon[30237]: osdmap e721: 8 total, 8 up, 8 in 2024-04-27T01:26:01.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:26:01.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:00 smithi162 ceph-mon[37680]: pgmap v1113: 172 pgs: 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:26:01.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:01.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:00 smithi162 ceph-mon[37680]: osdmap e721: 8 total, 8 up, 8 in 2024-04-27T01:26:01.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:26:01.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:00 smithi162 ceph-mon[26714]: pgmap v1113: 172 pgs: 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:26:01.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:01.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:00 smithi162 ceph-mon[26714]: osdmap e721: 8 total, 8 up, 8 in 2024-04-27T01:26:01.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:26:03.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:26:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:26:03.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:02 smithi181 ceph-mon[30237]: pgmap v1116: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:26:03.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:26:03.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:02 smithi181 ceph-mon[30237]: osdmap e722: 8 total, 8 up, 8 in 2024-04-27T01:26:03.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-144"}]: dispatch 2024-04-27T01:26:03.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:02 smithi162 ceph-mon[37680]: pgmap v1116: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:26:03.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:26:03.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:02 smithi162 ceph-mon[37680]: osdmap e722: 8 total, 8 up, 8 in 2024-04-27T01:26:03.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-144"}]: dispatch 2024-04-27T01:26:03.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:02 smithi162 ceph-mon[26714]: pgmap v1116: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 733 MiB used, 715 GiB / 715 GiB avail 2024-04-27T01:26:03.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:26:03.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:02 smithi162 ceph-mon[26714]: osdmap e722: 8 total, 8 up, 8 in 2024-04-27T01:26:03.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-144"}]: dispatch 2024-04-27T01:26:04.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-144"}]': finished 2024-04-27T01:26:04.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:03 smithi181 ceph-mon[30237]: osdmap e723: 8 total, 8 up, 8 in 2024-04-27T01:26:04.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-144", "mode": "readproxy"}]: dispatch 2024-04-27T01:26:04.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:04.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:03 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:26:04.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:03 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-144", "mode": "readproxy"}]': finished 2024-04-27T01:26:04.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:03 smithi181 ceph-mon[30237]: osdmap e724: 8 total, 8 up, 8 in 2024-04-27T01:26:04.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-144"}]': finished 2024-04-27T01:26:04.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[37680]: osdmap e723: 8 total, 8 up, 8 in 2024-04-27T01:26:04.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-144", "mode": "readproxy"}]: dispatch 2024-04-27T01:26:04.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:04.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:26:04.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-144", "mode": "readproxy"}]': finished 2024-04-27T01:26:04.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[37680]: osdmap e724: 8 total, 8 up, 8 in 2024-04-27T01:26:04.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-144"}]': finished 2024-04-27T01:26:04.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[26714]: osdmap e723: 8 total, 8 up, 8 in 2024-04-27T01:26:04.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-144", "mode": "readproxy"}]: dispatch 2024-04-27T01:26:04.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:04.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:26:04.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-144", "mode": "readproxy"}]': finished 2024-04-27T01:26:04.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:03 smithi162 ceph-mon[26714]: osdmap e724: 8 total, 8 up, 8 in 2024-04-27T01:26:05.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:04 smithi181 ceph-mon[30237]: pgmap v1119: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:26:05.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:04 smithi162 ceph-mon[37680]: pgmap v1119: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:26:05.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:04 smithi162 ceph-mon[26714]: pgmap v1119: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:26:06.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:05 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:06.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:05 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:06.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:05 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:07.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:06 smithi181 ceph-mon[30237]: pgmap v1121: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 218 B/s wr, 1 op/s 2024-04-27T01:26:07.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:06 smithi162 ceph-mon[37680]: pgmap v1121: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 218 B/s wr, 1 op/s 2024-04-27T01:26:07.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:06 smithi162 ceph-mon[26714]: pgmap v1121: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 218 B/s wr, 1 op/s 2024-04-27T01:26:09.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:08 smithi181 ceph-mon[30237]: pgmap v1122: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-27T01:26:09.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:08 smithi162 ceph-mon[26714]: pgmap v1122: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-27T01:26:09.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:08 smithi162 ceph-mon[37680]: pgmap v1122: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-27T01:26:10.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:26:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:26:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:26:11.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:10 smithi181 ceph-mon[30237]: pgmap v1123: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 133 B/s wr, 1 op/s 2024-04-27T01:26:11.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:10 smithi162 ceph-mon[26714]: pgmap v1123: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 133 B/s wr, 1 op/s 2024-04-27T01:26:11.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:10 smithi162 ceph-mon[37680]: pgmap v1123: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 133 B/s wr, 1 op/s 2024-04-27T01:26:13.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:12 smithi181 ceph-mon[30237]: pgmap v1124: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-04-27T01:26:13.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:26:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:26:13.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:12 smithi162 ceph-mon[26714]: pgmap v1124: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-04-27T01:26:13.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:12 smithi162 ceph-mon[37680]: pgmap v1124: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-04-27T01:26:13.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:13.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:14.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:15.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:14 smithi181 ceph-mon[30237]: pgmap v1125: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-27T01:26:15.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:15.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:14 smithi162 ceph-mon[26714]: pgmap v1125: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-27T01:26:15.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:15.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:14 smithi162 ceph-mon[37680]: pgmap v1125: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-27T01:26:15.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:16.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:16.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:15 smithi181 ceph-mon[30237]: osdmap e725: 8 total, 8 up, 8 in 2024-04-27T01:26:16.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144"}]: dispatch 2024-04-27T01:26:16.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:16.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:15 smithi162 ceph-mon[26714]: osdmap e725: 8 total, 8 up, 8 in 2024-04-27T01:26:16.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144"}]: dispatch 2024-04-27T01:26:16.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:16.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:15 smithi162 ceph-mon[37680]: osdmap e725: 8 total, 8 up, 8 in 2024-04-27T01:26:16.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144"}]: dispatch 2024-04-27T01:26:17.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:17 smithi181 ceph-mon[30237]: pgmap v1127: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-27T01:26:17.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:17 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:26:17.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144"}]': finished 2024-04-27T01:26:17.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:17 smithi181 ceph-mon[30237]: osdmap e726: 8 total, 8 up, 8 in 2024-04-27T01:26:17.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:17.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144"}]: dispatch 2024-04-27T01:26:17.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[26714]: pgmap v1127: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-27T01:26:17.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:26:17.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144"}]': finished 2024-04-27T01:26:17.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[26714]: osdmap e726: 8 total, 8 up, 8 in 2024-04-27T01:26:17.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:17.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144"}]: dispatch 2024-04-27T01:26:17.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[37680]: pgmap v1127: 204 pgs: 204 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-27T01:26:17.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:26:17.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144"}]': finished 2024-04-27T01:26:17.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[37680]: osdmap e726: 8 total, 8 up, 8 in 2024-04-27T01:26:17.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:17.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-144"}]: dispatch 2024-04-27T01:26:18.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:18 smithi181 ceph-mon[30237]: osdmap e727: 8 total, 8 up, 8 in 2024-04-27T01:26:18.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:18 smithi181 ceph-mon[30237]: osdmap e728: 8 total, 8 up, 8 in 2024-04-27T01:26:18.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:18.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:18 smithi162 ceph-mon[37680]: osdmap e727: 8 total, 8 up, 8 in 2024-04-27T01:26:18.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:18 smithi162 ceph-mon[37680]: osdmap e728: 8 total, 8 up, 8 in 2024-04-27T01:26:18.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:18.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:18 smithi162 ceph-mon[26714]: osdmap e727: 8 total, 8 up, 8 in 2024-04-27T01:26:18.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:18 smithi162 ceph-mon[26714]: osdmap e728: 8 total, 8 up, 8 in 2024-04-27T01:26:18.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:19.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:19 smithi181 ceph-mon[30237]: pgmap v1130: 172 pgs: 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:26:19.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:19.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:19 smithi181 ceph-mon[30237]: osdmap e729: 8 total, 8 up, 8 in 2024-04-27T01:26:19.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:19 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:26:19.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:19 smithi162 ceph-mon[26714]: pgmap v1130: 172 pgs: 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:26:19.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:19.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:19 smithi162 ceph-mon[26714]: osdmap e729: 8 total, 8 up, 8 in 2024-04-27T01:26:19.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:19 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:26:19.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:19 smithi162 ceph-mon[37680]: pgmap v1130: 172 pgs: 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-27T01:26:19.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:19.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:19 smithi162 ceph-mon[37680]: osdmap e729: 8 total, 8 up, 8 in 2024-04-27T01:26:19.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:19 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:26:20.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:26:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:26:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:26:21.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:20 smithi181 ceph-mon[30237]: pgmap v1133: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:26:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:26:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:20 smithi181 ceph-mon[30237]: osdmap e730: 8 total, 8 up, 8 in 2024-04-27T01:26:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:20 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-146"}]: dispatch 2024-04-27T01:26:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:20 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:21.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:21.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[26714]: pgmap v1133: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:26:21.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:26:21.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[26714]: osdmap e730: 8 total, 8 up, 8 in 2024-04-27T01:26:21.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-146"}]: dispatch 2024-04-27T01:26:21.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:21.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[37680]: pgmap v1133: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:26:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:26:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[37680]: osdmap e730: 8 total, 8 up, 8 in 2024-04-27T01:26:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-146"}]: dispatch 2024-04-27T01:26:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:21.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:22.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-146"}]': finished 2024-04-27T01:26:22.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[37680]: osdmap e731: 8 total, 8 up, 8 in 2024-04-27T01:26:22.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-146", "mode": "writeback"}]: dispatch 2024-04-27T01:26:22.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:26:22.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-146", "mode": "writeback"}]': finished 2024-04-27T01:26:22.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[37680]: osdmap e732: 8 total, 8 up, 8 in 2024-04-27T01:26:22.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-146"}]': finished 2024-04-27T01:26:22.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[26714]: osdmap e731: 8 total, 8 up, 8 in 2024-04-27T01:26:22.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-146", "mode": "writeback"}]: dispatch 2024-04-27T01:26:22.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:26:22.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-146", "mode": "writeback"}]': finished 2024-04-27T01:26:22.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:21 smithi162 ceph-mon[26714]: osdmap e732: 8 total, 8 up, 8 in 2024-04-27T01:26:22.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi162-74551-111", "overlaypool": "test-rados-api-smithi162-74551-146"}]': finished 2024-04-27T01:26:22.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:21 smithi181 ceph-mon[30237]: osdmap e731: 8 total, 8 up, 8 in 2024-04-27T01:26:22.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-146", "mode": "writeback"}]: dispatch 2024-04-27T01:26:22.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:21 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-27T01:26:22.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:21 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi162-74551-146", "mode": "writeback"}]': finished 2024-04-27T01:26:22.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:21 smithi181 ceph-mon[30237]: osdmap e732: 8 total, 8 up, 8 in 2024-04-27T01:26:23.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:22 smithi162 ceph-mon[37680]: pgmap v1136: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:23.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:22 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:26:23.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:22 smithi162 ceph-mon[26714]: pgmap v1136: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:23.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:22 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:26:23.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:22 smithi181 ceph-mon[30237]: pgmap v1136: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:23.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:22 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-27T01:26:23.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:26:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:26:24.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:26:24.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:23 smithi162 ceph-mon[37680]: osdmap e733: 8 total, 8 up, 8 in 2024-04-27T01:26:24.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:23 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:26:24.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:24.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:26:24.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:23 smithi162 ceph-mon[26714]: osdmap e733: 8 total, 8 up, 8 in 2024-04-27T01:26:24.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:23 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:26:24.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:24.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_count","val": "2"}]': finished 2024-04-27T01:26:24.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:23 smithi181 ceph-mon[30237]: osdmap e733: 8 total, 8 up, 8 in 2024-04-27T01:26:24.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:23 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-27T01:26:24.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:25.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:24 smithi162 ceph-mon[26714]: pgmap v1139: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:26:25.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:26:25.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:24 smithi162 ceph-mon[26714]: osdmap e734: 8 total, 8 up, 8 in 2024-04-27T01:26:25.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:24 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:26:25.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:24 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:26:25.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:24 smithi162 ceph-mon[37680]: pgmap v1139: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:26:25.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:26:25.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:24 smithi162 ceph-mon[37680]: osdmap e734: 8 total, 8 up, 8 in 2024-04-27T01:26:25.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:24 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:26:25.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:24 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:26:25.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:24 smithi181 ceph-mon[30237]: pgmap v1139: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:26:25.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_period","val": "600"}]': finished 2024-04-27T01:26:25.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:24 smithi181 ceph-mon[30237]: osdmap e734: 8 total, 8 up, 8 in 2024-04-27T01:26:25.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:24 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-27T01:26:25.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:24 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:26:26.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:26:26.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:26:26.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[37680]: osdmap e735: 8 total, 8 up, 8 in 2024-04-27T01:26:26.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:26:26.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:26.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[26714]: osdmap e735: 8 total, 8 up, 8 in 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:26.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:26:26.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:26:26.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:26.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:25 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:26:26.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:25 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-27T01:26:26.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-27T01:26:26.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:25 smithi181 ceph-mon[30237]: osdmap e735: 8 total, 8 up, 8 in 2024-04-27T01:26:26.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-27T01:26:26.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:25 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:26.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:25 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:26.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:25 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:26:26.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:25 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:26:26.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:25 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:26.258 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:25 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-27T01:26:27.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:26 smithi162 ceph-mon[37680]: pgmap v1142: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:26:27.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:26 smithi162 ceph-mon[37680]: osdmap e736: 8 total, 8 up, 8 in 2024-04-27T01:26:27.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-27T01:26:27.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:26 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "target_max_objects","val": "1"}]': finished 2024-04-27T01:26:27.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:26 smithi162 ceph-mon[37680]: osdmap e737: 8 total, 8 up, 8 in 2024-04-27T01:26:27.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:26 smithi162 ceph-mon[26714]: pgmap v1142: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:26:27.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:26 smithi162 ceph-mon[26714]: osdmap e736: 8 total, 8 up, 8 in 2024-04-27T01:26:27.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-27T01:26:27.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:26 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "target_max_objects","val": "1"}]': finished 2024-04-27T01:26:27.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:26 smithi162 ceph-mon[26714]: osdmap e737: 8 total, 8 up, 8 in 2024-04-27T01:26:27.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:26 smithi181 ceph-mon[30237]: pgmap v1142: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-27T01:26:27.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:26 smithi181 ceph-mon[30237]: osdmap e736: 8 total, 8 up, 8 in 2024-04-27T01:26:27.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-27T01:26:27.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:26 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-146","var": "target_max_objects","val": "1"}]': finished 2024-04-27T01:26:27.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:26 smithi181 ceph-mon[30237]: osdmap e737: 8 total, 8 up, 8 in 2024-04-27T01:26:28.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:27 smithi181 ceph-mon[30237]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-27T01:26:28.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:27 smithi162 ceph-mon[37680]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-27T01:26:28.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:27 smithi162 ceph-mon[26714]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-27T01:26:29.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:28 smithi181 ceph-mon[30237]: pgmap v1145: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:29.007 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:26:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=sqlstore.transactions t=2024-04-27T01:26:28.658385943Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-27T01:26:29.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:28 smithi162 ceph-mon[37680]: pgmap v1145: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:29.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:28 smithi162 ceph-mon[26714]: pgmap v1145: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:30.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:26:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:26:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:26:31.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:30 smithi181 ceph-mon[30237]: pgmap v1146: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 913 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:31.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:30 smithi162 ceph-mon[37680]: pgmap v1146: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 913 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:31.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:30 smithi162 ceph-mon[26714]: pgmap v1146: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 913 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:33.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:26:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:26:33.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:32 smithi181 ceph-mon[30237]: pgmap v1147: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:33.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:32 smithi162 ceph-mon[26714]: pgmap v1147: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:33.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:32 smithi162 ceph-mon[37680]: pgmap v1147: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:34.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:34.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:34.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:35.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:34 smithi181 ceph-mon[30237]: pgmap v1148: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:35.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:34 smithi162 ceph-mon[37680]: pgmap v1148: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:35.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:34 smithi162 ceph-mon[26714]: pgmap v1148: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:37.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:36 smithi181 ceph-mon[30237]: pgmap v1149: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:37.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:37.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:37.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:36 smithi162 ceph-mon[37680]: pgmap v1149: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:37.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:37.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:37.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:36 smithi162 ceph-mon[26714]: pgmap v1149: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:37.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:37.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:38.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:37 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:38.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:37 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:38.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:37 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:39.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:38 smithi181 ceph-mon[30237]: pgmap v1150: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:39.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:39.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:38 smithi181 ceph-mon[30237]: osdmap e738: 8 total, 8 up, 8 in 2024-04-27T01:26:39.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:38 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146"}]: dispatch 2024-04-27T01:26:39.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:38 smithi162 ceph-mon[37680]: pgmap v1150: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:39.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:39.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:38 smithi162 ceph-mon[37680]: osdmap e738: 8 total, 8 up, 8 in 2024-04-27T01:26:39.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:38 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146"}]: dispatch 2024-04-27T01:26:39.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:38 smithi162 ceph-mon[26714]: pgmap v1150: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-27T01:26:39.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:39.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:38 smithi162 ceph-mon[26714]: osdmap e738: 8 total, 8 up, 8 in 2024-04-27T01:26:39.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:38 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146"}]: dispatch 2024-04-27T01:26:40.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146"}]': finished 2024-04-27T01:26:40.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:39 smithi181 ceph-mon[30237]: osdmap e739: 8 total, 8 up, 8 in 2024-04-27T01:26:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:40.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:39 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146"}]: dispatch 2024-04-27T01:26:40.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:26:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:26:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:26:40.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146"}]': finished 2024-04-27T01:26:40.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:39 smithi162 ceph-mon[26714]: osdmap e739: 8 total, 8 up, 8 in 2024-04-27T01:26:40.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:40.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:39 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146"}]: dispatch 2024-04-27T01:26:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146"}]': finished 2024-04-27T01:26:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:39 smithi162 ceph-mon[37680]: osdmap e739: 8 total, 8 up, 8 in 2024-04-27T01:26:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:40.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:39 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-146"}]: dispatch 2024-04-27T01:26:41.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:40 smithi181 ceph-mon[30237]: pgmap v1153: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:26:41.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:40 smithi181 ceph-mon[30237]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:26:41.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:40 smithi181 ceph-mon[30237]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-27T01:26:41.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:40 smithi181 ceph-mon[30237]: osdmap e740: 8 total, 8 up, 8 in 2024-04-27T01:26:41.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:40 smithi162 ceph-mon[37680]: pgmap v1153: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:26:41.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:40 smithi162 ceph-mon[37680]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:26:41.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:40 smithi162 ceph-mon[37680]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-27T01:26:41.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:40 smithi162 ceph-mon[37680]: osdmap e740: 8 total, 8 up, 8 in 2024-04-27T01:26:41.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:40 smithi162 ceph-mon[26714]: pgmap v1153: 204 pgs: 204 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-27T01:26:41.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:40 smithi162 ceph-mon[26714]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:26:41.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:40 smithi162 ceph-mon[26714]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-27T01:26:41.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:40 smithi162 ceph-mon[26714]: osdmap e740: 8 total, 8 up, 8 in 2024-04-27T01:26:42.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:41 smithi181 ceph-mon[30237]: osdmap e741: 8 total, 8 up, 8 in 2024-04-27T01:26:42.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:41 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:42.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:41 smithi162 ceph-mon[37680]: osdmap e741: 8 total, 8 up, 8 in 2024-04-27T01:26:42.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:41 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:42.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:41 smithi162 ceph-mon[26714]: osdmap e741: 8 total, 8 up, 8 in 2024-04-27T01:26:42.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:41 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:42.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:42 smithi181 ceph-mon[30237]: pgmap v1156: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:42.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:42.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:42 smithi181 ceph-mon[30237]: osdmap e742: 8 total, 8 up, 8 in 2024-04-27T01:26:42.894 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:42 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:26:43.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:42 smithi162 ceph-mon[37680]: pgmap v1156: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:43.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:43.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:42 smithi162 ceph-mon[37680]: osdmap e742: 8 total, 8 up, 8 in 2024-04-27T01:26:43.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:42 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:26:43.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:42 smithi162 ceph-mon[26714]: pgmap v1156: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 805 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:43.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:43.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:42 smithi162 ceph-mon[26714]: osdmap e742: 8 total, 8 up, 8 in 2024-04-27T01:26:43.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:42 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-27T01:26:43.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:26:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:26:43.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:26:43.986 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:43 smithi162 ceph-mon[37680]: osdmap e743: 8 total, 8 up, 8 in 2024-04-27T01:26:43.986 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:43 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148"}]: dispatch 2024-04-27T01:26:43.986 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:43.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:26:43.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:43 smithi162 ceph-mon[26714]: osdmap e743: 8 total, 8 up, 8 in 2024-04-27T01:26:43.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:43 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148"}]: dispatch 2024-04-27T01:26:43.986 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:44.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-27T01:26:44.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:43 smithi181 ceph-mon[30237]: osdmap e743: 8 total, 8 up, 8 in 2024-04-27T01:26:44.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:43 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148"}]: dispatch 2024-04-27T01:26:44.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:45.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:44 smithi181 ceph-mon[30237]: pgmap v1159: 204 pgs: 204 active+clean; 4.4 MiB data, 806 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:45.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148"}]': finished 2024-04-27T01:26:45.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:44 smithi181 ceph-mon[30237]: osdmap e744: 8 total, 8 up, 8 in 2024-04-27T01:26:45.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:45.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:44 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148"}]: dispatch 2024-04-27T01:26:45.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:44 smithi162 ceph-mon[37680]: pgmap v1159: 204 pgs: 204 active+clean; 4.4 MiB data, 806 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:45.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148"}]': finished 2024-04-27T01:26:45.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:44 smithi162 ceph-mon[37680]: osdmap e744: 8 total, 8 up, 8 in 2024-04-27T01:26:45.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:45.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:44 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148"}]: dispatch 2024-04-27T01:26:45.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:44 smithi162 ceph-mon[26714]: pgmap v1159: 204 pgs: 204 active+clean; 4.4 MiB data, 806 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:45.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148"}]': finished 2024-04-27T01:26:45.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:44 smithi162 ceph-mon[26714]: osdmap e744: 8 total, 8 up, 8 in 2024-04-27T01:26:45.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:45.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:44 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-148"}]: dispatch 2024-04-27T01:26:46.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:45 smithi181 ceph-mon[30237]: osdmap e745: 8 total, 8 up, 8 in 2024-04-27T01:26:46.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:45 smithi181 ceph-mon[30237]: osdmap e746: 8 total, 8 up, 8 in 2024-04-27T01:26:46.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:45 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:46.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:45 smithi162 ceph-mon[37680]: osdmap e745: 8 total, 8 up, 8 in 2024-04-27T01:26:46.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:45 smithi162 ceph-mon[37680]: osdmap e746: 8 total, 8 up, 8 in 2024-04-27T01:26:46.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:45 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:46.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:45 smithi162 ceph-mon[26714]: osdmap e745: 8 total, 8 up, 8 in 2024-04-27T01:26:46.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:45 smithi162 ceph-mon[26714]: osdmap e746: 8 total, 8 up, 8 in 2024-04-27T01:26:46.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:45 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:47.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:46 smithi181 ceph-mon[30237]: pgmap v1162: 172 pgs: 172 active+clean; 4.4 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:26:47.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:47.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:46 smithi181 ceph-mon[30237]: osdmap e747: 8 total, 8 up, 8 in 2024-04-27T01:26:47.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:47.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:46 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-150"}]: dispatch 2024-04-27T01:26:47.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:46 smithi162 ceph-mon[37680]: pgmap v1162: 172 pgs: 172 active+clean; 4.4 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:26:47.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:47.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:46 smithi162 ceph-mon[37680]: osdmap e747: 8 total, 8 up, 8 in 2024-04-27T01:26:47.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:47.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:46 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-150"}]: dispatch 2024-04-27T01:26:47.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:46 smithi162 ceph-mon[26714]: pgmap v1162: 172 pgs: 172 active+clean; 4.4 MiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-27T01:26:47.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:47.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:46 smithi162 ceph-mon[26714]: osdmap e747: 8 total, 8 up, 8 in 2024-04-27T01:26:47.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:47.336 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:46 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-150"}]: dispatch 2024-04-27T01:26:49.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:48 smithi181 ceph-mon[30237]: pgmap v1165: 204 pgs: 204 active+clean; 4.4 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-27T01:26:49.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:48 smithi181 ceph-mon[30237]: osdmap e748: 8 total, 8 up, 8 in 2024-04-27T01:26:49.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:48 smithi162 ceph-mon[37680]: pgmap v1165: 204 pgs: 204 active+clean; 4.4 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-27T01:26:49.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:48 smithi162 ceph-mon[37680]: osdmap e748: 8 total, 8 up, 8 in 2024-04-27T01:26:49.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:48 smithi162 ceph-mon[26714]: pgmap v1165: 204 pgs: 204 active+clean; 4.4 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-27T01:26:49.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:48 smithi162 ceph-mon[26714]: osdmap e748: 8 total, 8 up, 8 in 2024-04-27T01:26:50.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:49 smithi181 ceph-mon[30237]: osdmap e749: 8 total, 8 up, 8 in 2024-04-27T01:26:50.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:49 smithi181 ceph-mon[30237]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:26:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:50.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:49 smithi181 ceph-mon[30237]: osdmap e750: 8 total, 8 up, 8 in 2024-04-27T01:26:50.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:26:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:26:50.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-mon[37680]: osdmap e749: 8 total, 8 up, 8 in 2024-04-27T01:26:50.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-mon[37680]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:26:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:50.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-mon[37680]: osdmap e750: 8 total, 8 up, 8 in 2024-04-27T01:26:50.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-mon[26714]: osdmap e749: 8 total, 8 up, 8 in 2024-04-27T01:26:50.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:50.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-mon[26714]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:26:50.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:50.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:49 smithi162 ceph-mon[26714]: osdmap e750: 8 total, 8 up, 8 in 2024-04-27T01:26:51.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:50 smithi181 ceph-mon[30237]: pgmap v1168: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:26:51.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:51.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-152"}]: dispatch 2024-04-27T01:26:51.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:50 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:51.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:50 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:51.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:50 smithi181 ceph-mon[30237]: osdmap e751: 8 total, 8 up, 8 in 2024-04-27T01:26:51.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[37680]: pgmap v1168: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:26:51.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:51.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-152"}]: dispatch 2024-04-27T01:26:51.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:51.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:51.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[37680]: osdmap e751: 8 total, 8 up, 8 in 2024-04-27T01:26:51.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[26714]: pgmap v1168: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 810 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-27T01:26:51.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:51.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-152"}]: dispatch 2024-04-27T01:26:51.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:26:51.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:26:51.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:50 smithi162 ceph-mon[26714]: osdmap e751: 8 total, 8 up, 8 in 2024-04-27T01:26:53.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:26:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:26:53.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:52 smithi181 ceph-mon[30237]: pgmap v1171: 172 pgs: 172 active+clean; 4.4 MiB data, 810 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:53.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:52 smithi181 ceph-mon[30237]: osdmap e752: 8 total, 8 up, 8 in 2024-04-27T01:26:53.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:52 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:53.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:52 smithi162 ceph-mon[37680]: pgmap v1171: 172 pgs: 172 active+clean; 4.4 MiB data, 810 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:53.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:52 smithi162 ceph-mon[37680]: osdmap e752: 8 total, 8 up, 8 in 2024-04-27T01:26:53.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:52 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:53.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:52 smithi162 ceph-mon[26714]: pgmap v1171: 172 pgs: 172 active+clean; 4.4 MiB data, 810 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:52 smithi162 ceph-mon[26714]: osdmap e752: 8 total, 8 up, 8 in 2024-04-27T01:26:53.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:52 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-27T01:26:54.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:54.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:53 smithi181 ceph-mon[30237]: osdmap e753: 8 total, 8 up, 8 in 2024-04-27T01:26:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-111","var": "dedup_tier","val": "test-rados-api-smithi162-74551-154"}]: dispatch 2024-04-27T01:26:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:53 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-154"}]: dispatch 2024-04-27T01:26:54.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:54.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:54.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[37680]: osdmap e753: 8 total, 8 up, 8 in 2024-04-27T01:26:54.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-111","var": "dedup_tier","val": "test-rados-api-smithi162-74551-154"}]: dispatch 2024-04-27T01:26:54.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-154"}]: dispatch 2024-04-27T01:26:54.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi162-74551-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-27T01:26:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[26714]: osdmap e753: 8 total, 8 up, 8 in 2024-04-27T01:26:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi162-74551-111","var": "dedup_tier","val": "test-rados-api-smithi162-74551-154"}]: dispatch 2024-04-27T01:26:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi162-74551-111", "tierpool": "test-rados-api-smithi162-74551-154"}]: dispatch 2024-04-27T01:26:54.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:26:55.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:54 smithi181 ceph-mon[30237]: pgmap v1174: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-27T01:26:55.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:54 smithi181 ceph-mon[30237]: osdmap e754: 8 total, 8 up, 8 in 2024-04-27T01:26:55.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:54 smithi162 ceph-mon[37680]: pgmap v1174: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-27T01:26:55.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:54 smithi162 ceph-mon[37680]: osdmap e754: 8 total, 8 up, 8 in 2024-04-27T01:26:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:54 smithi162 ceph-mon[26714]: pgmap v1174: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-27T01:26:55.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:54 smithi162 ceph-mon[26714]: osdmap e754: 8 total, 8 up, 8 in 2024-04-27T01:26:56.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:55 smithi181 ceph-mon[30237]: osdmap e755: 8 total, 8 up, 8 in 2024-04-27T01:26:56.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:56.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:56.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:55 smithi181 ceph-mon[30237]: osdmap e756: 8 total, 8 up, 8 in 2024-04-27T01:26:56.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:55 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:56.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:55 smithi162 ceph-mon[37680]: osdmap e755: 8 total, 8 up, 8 in 2024-04-27T01:26:56.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:56.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:56.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:55 smithi162 ceph-mon[37680]: osdmap e756: 8 total, 8 up, 8 in 2024-04-27T01:26:56.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:55 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:56.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:55 smithi162 ceph-mon[26714]: osdmap e755: 8 total, 8 up, 8 in 2024-04-27T01:26:56.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:56.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:56.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:55 smithi162 ceph-mon[26714]: osdmap e756: 8 total, 8 up, 8 in 2024-04-27T01:26:56.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:55 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-111"}]: dispatch 2024-04-27T01:26:56.727 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7908 ms) 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12125 ms) 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8037 ms) 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12786 ms) 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8039 ms) 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8039 ms) 2024-04-27T01:26:56.728 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: ok, hit_set contains 337:602f83fe:::foo:head 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8162 ms) 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,0 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: first is 1714181131 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,0 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,0 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,1714181133,1714181134,0 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,1714181133,1714181134,0 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,1714181133,1714181134,0 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,1714181133,1714181134,1714181136,1714181137,0 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,1714181133,1714181134,1714181136,1714181137,0 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,1714181133,1714181134,1714181136,1714181137,0 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,1714181133,1714181134,1714181136,1714181137,1714181139,1714181140,0 2024-04-27T01:26:56.729 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,1714181133,1714181134,1714181136,1714181137,1714181139,1714181140,0 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181131,1714181133,1714181134,1714181136,1714181137,1714181139,1714181140,0 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: got ls 1714181134,1714181136,1714181137,1714181139,1714181140,1714181142,1714181143,0 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: first now 1714181134, trimmed 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21066 ms) 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: foo0 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14066 ms) 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18064 ms) 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22885 ms) 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-04-27T01:26:56.730 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5029 ms) 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3009 ms) 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3016 ms) 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3028 ms) 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (230971 ms total) 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (863961 ms total) 2024-04-27T01:26:56.731 INFO:tasks.workunit.client.0.smithi162.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74287 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74287 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74446 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74446 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75051 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75051 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74712 2024-04-27T01:26:56.737 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74712 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74556 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74556 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74364 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74364 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74593 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74593 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75246 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75246 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.738 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75316 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75316 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74395 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74395 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75101 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75101 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74925 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74925 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74259 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74259 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74327 2024-04-27T01:26:56.739 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74327 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74981 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74981 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74250 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74250 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74271 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74271 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75445 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75445 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74481 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74481 2024-04-27T01:26:56.740 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75529 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75529 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74576 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74576 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75221 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75221 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75573 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75573 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.741 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74534 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74534 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75018 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75018 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75397 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75397 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75197 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75197 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75480 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75480 2024-04-27T01:26:56.742 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74894 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74894 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74425 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74425 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75553 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75553 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74756 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74756 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74243 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74243 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.743 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74304 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74304 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74640 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74640 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74345 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74345 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74803 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74803 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=74844 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 74844 2024-04-27T01:26:56.744 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75329 2024-04-27T01:26:56.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75329 2024-04-27T01:26:56.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ for t in "${!pids[@]}" 2024-04-27T01:26:56.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=75596 2024-04-27T01:26:56.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 75596 2024-04-27T01:26:56.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ exit 0 2024-04-27T01:26:56.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ cleanup 2024-04-27T01:26:56.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ pkill -P 74237 2024-04-27T01:26:56.756 INFO:tasks.workunit.client.0.smithi162.stderr:+ true 2024-04-27T01:26:56.756 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-27T01:26:56.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-27T01:26:56.786 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-04-27T01:26:56.787 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-04-27T01:26:56.788 DEBUG:teuthology.orchestra.run.smithi162:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b22e2ebdeb24376882b7bda2a7329c8cccc2276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-04-27T01:26:56.847 INFO:tasks.workunit.client.0.smithi162.stderr:++ uuidgen 2024-04-27T01:26:56.851 INFO:tasks.workunit.client.0.smithi162.stderr:+ p=82d3730a-02a2-4166-9bba-a35fc501ddd8 2024-04-27T01:26:56.851 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool create 82d3730a-02a2-4166-9bba-a35fc501ddd8 12 2024-04-27T01:26:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.951+0000 7f8738faf640 1 -- 172.21.15.162:0/3310519198 >> v1:172.21.15.162:6790/0 conn(0x7f87341065b0 legacy=0x7f8734106990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.951+0000 7f8738faf640 1 -- 172.21.15.162:0/3310519198 shutdown_connections 2024-04-27T01:26:56.937 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.951+0000 7f8738faf640 1 -- 172.21.15.162:0/3310519198 >> 172.21.15.162:0/3310519198 conn(0x7f87340771a0 msgr2=0x7f87340fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:26:56.937 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.951+0000 7f8738faf640 1 -- 172.21.15.162:0/3310519198 shutdown_connections 2024-04-27T01:26:56.937 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.951+0000 7f8738faf640 1 -- 172.21.15.162:0/3310519198 wait complete. 2024-04-27T01:26:56.939 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.952+0000 7f8738faf640 1 Processor -- start 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.952+0000 7f8738faf640 1 -- start start 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.952+0000 7f8738faf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87341a66a0 con 0x7f87341065b0 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.952+0000 7f8738faf640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87341a78e0 con 0x7f8734109350 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.952+0000 7f8738faf640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87341a8b20 con 0x7f873410cf50 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.952+0000 7f873259c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f87341065b0 0x7f8734105990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:43436/0 (socket says 172.21.15.162:43436) 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.952+0000 7f873259c640 1 -- 172.21.15.162:0/4184148007 learned_addr learned my addr 172.21.15.162:0/4184148007 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.952+0000 7f8732d9d640 1 --1- 172.21.15.162:0/4184148007 >> v1:172.21.15.162:6790/0 conn(0x7f873410cf50 0x7f87341a4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:45392/0 (socket says 172.21.15.162:45392) 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2813678110 0 0) 0x7f87341a8b20 con 0x7f873410cf50 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f871c003520 con 0x7f873410cf50 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 182196696 0 0) 0x7f87341a66a0 con 0x7f87341065b0 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87341a8b20 con 0x7f87341065b0 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1219955902 0 0) 0x7f87341a78e0 con 0x7f8734109350 2024-04-27T01:26:56.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87341a66a0 con 0x7f8734109350 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 595647505 0 0) 0x7f87341a8b20 con 0x7f87341065b0 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87341a78e0 con 0x7f87341065b0 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8724001510 con 0x7f87341065b0 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2505581685 0 0) 0x7f871c003520 con 0x7f873410cf50 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87341a8b20 con 0x7f873410cf50 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f872c001b70 con 0x7f873410cf50 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3999039468 0 0) 0x7f87341a66a0 con 0x7f8734109350 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f871c003520 con 0x7f8734109350 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f87280018b0 con 0x7f8734109350 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1679600548 0 0) 0x7f87341a78e0 con 0x7f87341065b0 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 >> v1:172.21.15.162:6790/0 conn(0x7f873410cf50 legacy=0x7f87341a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 >> v1:172.21.15.181:6789/0 conn(0x7f8734109350 legacy=0x7f87341a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87341a9d60 con 0x7f87341065b0 2024-04-27T01:26:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f87341a68d0 con 0x7f87341065b0 2024-04-27T01:26:56.942 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f87341a6da0 con 0x7f87341065b0 2024-04-27T01:26:56.942 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f8724001af0 con 0x7f87341065b0 2024-04-27T01:26:56.942 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.953+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8724004ce0 con 0x7f87341065b0 2024-04-27T01:26:56.942 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.954+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87140050b0 con 0x7f87341065b0 2024-04-27T01:26:56.942 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.955+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f87240037d0 con 0x7f87341065b0 2024-04-27T01:26:56.942 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.955+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(757..757 src has 1..757) ==== 7366+0+0 (unknown 505248610 0 0) 0x7f8724089700 con 0x7f87341065b0 2024-04-27T01:26:56.944 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:56.960+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f8724056af0 con 0x7f87341065b0 2024-04-27T01:26:57.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:56 smithi181 ceph-mon[30237]: pgmap v1177: 164 pgs: 164 active+clean; 583 KiB data, 795 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:57.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:56 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:26:57.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:56 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:57.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:56 smithi181 ceph-mon[30237]: osdmap e757: 8 total, 8 up, 8 in 2024-04-27T01:26:57.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:56 smithi162 ceph-mon[37680]: pgmap v1177: 164 pgs: 164 active+clean; 583 KiB data, 795 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:57.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:56 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:26:57.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:56 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:57.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:56 smithi162 ceph-mon[37680]: osdmap e757: 8 total, 8 up, 8 in 2024-04-27T01:26:57.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:56 smithi162 ceph-mon[26714]: pgmap v1177: 164 pgs: 164 active+clean; 583 KiB data, 795 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:57.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:56 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:26:57.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:56 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/280230854' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi162-74551-111"}]': finished 2024-04-27T01:26:57.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:56 smithi162 ceph-mon[26714]: osdmap e757: 8 total, 8 up, 8 in 2024-04-27T01:26:57.124 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.139+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12} v 0) -- 0x7f87140053a0 con 0x7f87341065b0 2024-04-27T01:26:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.750+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]=0 pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' created v758) ==== 176+0+0 (unknown 2029845295 0 0) 0x7f872405a7a0 con 0x7f87341065b0 2024-04-27T01:26:57.846 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.861+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12} v 0) -- 0x7f87140058b0 con 0x7f87341065b0 2024-04-27T01:26:57.846 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.861+0000 7f87237fe640 1 -- 172.21.15.162:0/4184148007 <== mon.0 v1:172.21.15.162:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]=0 pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' already exists v758) ==== 183+0+0 (unknown 1545167976 0 0) 0x7f872405f060 con 0x7f87341065b0 2024-04-27T01:26:57.846 INFO:tasks.workunit.client.0.smithi162.stderr:pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' already exists 2024-04-27T01:26:57.848 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.863+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 >> v1:172.21.15.162:6800/624703537 conn(0x7f871c06eea0 legacy=0x7f871c071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:57.848 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.863+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 >> v1:172.21.15.162:6789/0 conn(0x7f87341065b0 legacy=0x7f8734105990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:57.848 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.863+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 shutdown_connections 2024-04-27T01:26:57.848 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.863+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 >> 172.21.15.162:0/4184148007 conn(0x7f87340771a0 msgr2=0x7f87340690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:26:57.848 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.863+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 shutdown_connections 2024-04-27T01:26:57.848 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.863+0000 7f8738faf640 1 -- 172.21.15.162:0/4184148007 wait complete. 2024-04-27T01:26:57.857 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool set-quota 82d3730a-02a2-4166-9bba-a35fc501ddd8 max_objects 10 2024-04-27T01:26:57.944 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.957+0000 7feab4702640 1 -- 172.21.15.162:0/3571698250 >> v1:172.21.15.162:6789/0 conn(0x7feaac1065a0 legacy=0x7feaac106980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:57.944 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.957+0000 7feab4702640 1 -- 172.21.15.162:0/3571698250 shutdown_connections 2024-04-27T01:26:57.944 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.957+0000 7feab4702640 1 -- 172.21.15.162:0/3571698250 >> 172.21.15.162:0/3571698250 conn(0x7feaac0fbfb0 msgr2=0x7feaac0fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.957+0000 7feab4702640 1 -- 172.21.15.162:0/3571698250 shutdown_connections 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.957+0000 7feab4702640 1 -- 172.21.15.162:0/3571698250 wait complete. 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.957+0000 7feab4702640 1 Processor -- start 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.957+0000 7feab4702640 1 -- start start 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.957+0000 7feab4702640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feaac1a5510 con 0x7feaac109340 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.957+0000 7feab4702640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feaac1a6750 con 0x7feaac10cf40 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.957+0000 7feab4702640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7feaac1a7990 con 0x7feaac1065a0 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feab1c9d640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7feaac109340 0x7feaac1a03b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:43450/0 (socket says 172.21.15.162:43450) 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feab1c9d640 1 -- 172.21.15.162:0/3667623701 learned_addr learned my addr 172.21.15.162:0/3667623701 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feab249e640 1 --1- 172.21.15.162:0/3667623701 >> v1:172.21.15.162:6790/0 conn(0x7feaac1065a0 0x7feaac1058b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:45416/0 (socket says 172.21.15.162:45416) 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1074899528 0 0) 0x7feaac1a5510 con 0x7feaac109340 2024-04-27T01:26:57.945 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea98003160 con 0x7feaac109340 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1936244107 0 0) 0x7feaac1a7990 con 0x7feaac1065a0 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaac1a5510 con 0x7feaac1065a0 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3371448791 0 0) 0x7feaac1a6750 con 0x7feaac10cf40 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaac1a7990 con 0x7feaac10cf40 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4218316062 0 0) 0x7feaac1a5510 con 0x7feaac1065a0 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feaac1a6750 con 0x7feaac1065a0 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7feaa80014f0 con 0x7feaac1065a0 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1648877817 0 0) 0x7fea98003160 con 0x7feaac109340 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feaac1a5510 con 0x7feaac109340 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fea9c0018b0 con 0x7feaac109340 2024-04-27T01:26:57.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1938163623 0 0) 0x7feaac1a7990 con 0x7feaac10cf40 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fea98003160 con 0x7feaac10cf40 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7feaa4001b40 con 0x7feaac10cf40 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2244725076 0 0) 0x7feaac1a6750 con 0x7feaac1065a0 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 >> v1:172.21.15.181:6789/0 conn(0x7feaac10cf40 legacy=0x7feaac1a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 >> v1:172.21.15.162:6789/0 conn(0x7feaac109340 legacy=0x7feaac1a03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac1a8bd0 con 0x7feaac1065a0 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feaac1a7bc0 con 0x7feaac1065a0 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7feaac1a8090 con 0x7feaac1065a0 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7feaa8001b40 con 0x7feaac1065a0 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.958+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7feaa8004d90 con 0x7feaac1065a0 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.959+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea90005080 con 0x7feaac1065a0 2024-04-27T01:26:57.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.960+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7feaa8005010 con 0x7feaac1065a0 2024-04-27T01:26:57.948 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.960+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(758..758 src has 1..758) ==== 7741+0+0 (unknown 4006493610 0 0) 0x7feaa8089ad0 con 0x7feaac1065a0 2024-04-27T01:26:57.950 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:57.965+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7feaa8056c70 con 0x7feaac1065a0 2024-04-27T01:26:58.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:57 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4184148007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]: dispatch 2024-04-27T01:26:58.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:57 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4184148007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]: dispatch 2024-04-27T01:26:58.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:57 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4184148007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]: dispatch 2024-04-27T01:26:58.131 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:58.147+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"} v 0) -- 0x7fea90005370 con 0x7feaac1065a0 2024-04-27T01:26:58.746 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:58.761+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v759) ==== 223+0+0 (unknown 1966705873 0 0) 0x7feaa805a920 con 0x7feaac1065a0 2024-04-27T01:26:58.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:58.874+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"} v 0) -- 0x7fea90001090 con 0x7feaac1065a0 2024-04-27T01:26:59.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:58 smithi181 ceph-mon[30237]: pgmap v1180: 164 pgs: 164 active+clean; 583 KiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:59.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4184148007' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]': finished 2024-04-27T01:26:59.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:58 smithi181 ceph-mon[30237]: osdmap e758: 8 total, 8 up, 8 in 2024-04-27T01:26:59.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4184148007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]: dispatch 2024-04-27T01:26:59.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:58 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3667623701' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:26:59.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:58 smithi181 ceph-mon[30237]: from='client.57071 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:26:59.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[37680]: pgmap v1180: 164 pgs: 164 active+clean; 583 KiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:59.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4184148007' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]': finished 2024-04-27T01:26:59.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[37680]: osdmap e758: 8 total, 8 up, 8 in 2024-04-27T01:26:59.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4184148007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]: dispatch 2024-04-27T01:26:59.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3667623701' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:26:59.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[37680]: from='client.57071 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:26:59.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[26714]: pgmap v1180: 164 pgs: 164 active+clean; 583 KiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:26:59.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4184148007' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]': finished 2024-04-27T01:26:59.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[26714]: osdmap e758: 8 total, 8 up, 8 in 2024-04-27T01:26:59.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4184148007' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pg_num": 12}]: dispatch 2024-04-27T01:26:59.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3667623701' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:26:59.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:58 smithi162 ceph-mon[26714]: from='client.57071 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:26:59.743 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.758+0000 7feaa37fe640 1 -- 172.21.15.162:0/3667623701 <== mon.2 v1:172.21.15.162:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v760) ==== 223+0+0 (unknown 3489154578 0 0) 0x7feaa805f1e0 con 0x7feaac1065a0 2024-04-27T01:26:59.743 INFO:tasks.workunit.client.0.smithi162.stderr:set-quota max_objects = 10 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 2024-04-27T01:26:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.762+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 >> v1:172.21.15.162:6800/624703537 conn(0x7fea9806eee0 legacy=0x7fea980713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.762+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 >> v1:172.21.15.162:6790/0 conn(0x7feaac1065a0 legacy=0x7feaac1058b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.762+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 shutdown_connections 2024-04-27T01:26:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.762+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 >> 172.21.15.162:0/3667623701 conn(0x7feaac0fbfb0 msgr2=0x7feaac0fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:26:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.762+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 shutdown_connections 2024-04-27T01:26:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.762+0000 7feab4702640 1 -- 172.21.15.162:0/3667623701 wait complete. 2024-04-27T01:26:59.758 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool application enable 82d3730a-02a2-4166-9bba-a35fc501ddd8 rados 2024-04-27T01:26:59.847 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.861+0000 7f1688aa4640 1 -- 172.21.15.162:0/2938397942 >> v1:172.21.15.162:6789/0 conn(0x7f16841065a0 legacy=0x7f1684106980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:59.847 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.861+0000 7f1688aa4640 1 -- 172.21.15.162:0/2938397942 shutdown_connections 2024-04-27T01:26:59.847 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.861+0000 7f1688aa4640 1 -- 172.21.15.162:0/2938397942 >> 172.21.15.162:0/2938397942 conn(0x7f16840fbfb0 msgr2=0x7f16840fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:26:59.847 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.861+0000 7f1688aa4640 1 -- 172.21.15.162:0/2938397942 shutdown_connections 2024-04-27T01:26:59.847 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.861+0000 7f1688aa4640 1 -- 172.21.15.162:0/2938397942 wait complete. 2024-04-27T01:26:59.847 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.862+0000 7f1688aa4640 1 Processor -- start 2024-04-27T01:26:59.847 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.862+0000 7f1688aa4640 1 -- start start 2024-04-27T01:26:59.851 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.862+0000 7f1688aa4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1684105be0 con 0x7f16841065a0 2024-04-27T01:26:59.851 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.862+0000 7f1688aa4640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f1684105e10 con 0x7f1684109340 2024-04-27T01:26:59.851 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.862+0000 7f1688aa4640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f16841a7b20 con 0x7f168410cf40 2024-04-27T01:26:59.852 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.862+0000 7f168259c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f16841065a0 0x7f1684102a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:43464/0 (socket says 172.21.15.162:43464) 2024-04-27T01:26:59.852 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.862+0000 7f168259c640 1 -- 172.21.15.162:0/1809297007 learned_addr learned my addr 172.21.15.162:0/1809297007 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:26:59.852 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.862+0000 7f1682d9d640 1 --1- 172.21.15.162:0/1809297007 >> v1:172.21.15.162:6790/0 conn(0x7f168410cf40 0x7f16841a43e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:45438/0 (socket says 172.21.15.162:45438) 2024-04-27T01:26:59.852 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2896603987 0 0) 0x7f1684105be0 con 0x7f16841065a0 2024-04-27T01:26:59.852 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1670003160 con 0x7f16841065a0 2024-04-27T01:26:59.852 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1787353038 0 0) 0x7f16841a7b20 con 0x7f168410cf40 2024-04-27T01:26:59.853 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1684105be0 con 0x7f168410cf40 2024-04-27T01:26:59.853 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3775816502 0 0) 0x7f1684105e10 con 0x7f1684109340 2024-04-27T01:26:59.853 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16841a7b20 con 0x7f1684109340 2024-04-27T01:26:59.853 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 387494741 0 0) 0x7f1670003160 con 0x7f16841065a0 2024-04-27T01:26:59.853 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1684105e10 con 0x7f16841065a0 2024-04-27T01:26:59.853 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1674004480 con 0x7f16841065a0 2024-04-27T01:26:59.854 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3396509449 0 0) 0x7f1684105be0 con 0x7f168410cf40 2024-04-27T01:26:59.854 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1670003160 con 0x7f168410cf40 2024-04-27T01:26:59.854 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f167c001c10 con 0x7f168410cf40 2024-04-27T01:26:59.854 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3683699508 0 0) 0x7f16841a7b20 con 0x7f1684109340 2024-04-27T01:26:59.854 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1684105be0 con 0x7f1684109340 2024-04-27T01:26:59.854 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f166c001830 con 0x7f1684109340 2024-04-27T01:26:59.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2600749918 0 0) 0x7f1684105e10 con 0x7f16841065a0 2024-04-27T01:26:59.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 >> v1:172.21.15.162:6790/0 conn(0x7f168410cf40 legacy=0x7f16841a43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:59.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 >> v1:172.21.15.181:6789/0 conn(0x7f1684109340 legacy=0x7f1684103170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:26:59.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16841a8d00 con 0x7f16841065a0 2024-04-27T01:26:59.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f16841a7d50 con 0x7f16841065a0 2024-04-27T01:26:59.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f16841a8220 con 0x7f16841065a0 2024-04-27T01:26:59.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f1674001ad0 con 0x7f16841065a0 2024-04-27T01:26:59.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.863+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f1674004d90 con 0x7f16841065a0 2024-04-27T01:26:59.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.864+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1664005080 con 0x7f16841065a0 2024-04-27T01:26:59.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.865+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f1674005010 con 0x7f16841065a0 2024-04-27T01:26:59.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.865+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(760..760 src has 1..760) ==== 7741+0+0 (unknown 3791924147 0 0) 0x7f1674089950 con 0x7f16841065a0 2024-04-27T01:26:59.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:26:59.870+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f1674056bd0 con 0x7f16841065a0 2024-04-27T01:27:00.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:00.050+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"} v 0) -- 0x7f1664005370 con 0x7f16841065a0 2024-04-27T01:27:00.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:26:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:27:00.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[37680]: from='client.57071 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:27:00.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[37680]: osdmap e759: 8 total, 8 up, 8 in 2024-04-27T01:27:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3667623701' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:27:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[37680]: from='client.57071 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:27:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[37680]: from='client.57071 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:27:00.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[37680]: osdmap e760: 8 total, 8 up, 8 in 2024-04-27T01:27:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[26714]: from='client.57071 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:27:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[26714]: osdmap e759: 8 total, 8 up, 8 in 2024-04-27T01:27:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3667623701' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:27:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[26714]: from='client.57071 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:27:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[26714]: from='client.57071 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:27:00.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:26:59 smithi162 ceph-mon[26714]: osdmap e760: 8 total, 8 up, 8 in 2024-04-27T01:27:00.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:59 smithi181 ceph-mon[30237]: from='client.57071 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:27:00.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:59 smithi181 ceph-mon[30237]: osdmap e759: 8 total, 8 up, 8 in 2024-04-27T01:27:00.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:59 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3667623701' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:27:00.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:59 smithi181 ceph-mon[30237]: from='client.57071 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:27:00.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:59 smithi181 ceph-mon[30237]: from='client.57071 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:27:00.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:26:59 smithi181 ceph-mon[30237]: osdmap e760: 8 total, 8 up, 8 in 2024-04-27T01:27:00.765 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:00.780+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]=0 enabled application 'rados' on pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' v761) ==== 213+0+0 (unknown 1721011901 0 0) 0x7f167405a880 con 0x7f16841065a0 2024-04-27T01:27:00.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:00.891+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"} v 0) -- 0x7f1664001090 con 0x7f16841065a0 2024-04-27T01:27:01.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:00 smithi162 ceph-mon[26714]: pgmap v1183: 176 pgs: 12 unknown, 164 active+clean; 583 KiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:01.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:00 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]: dispatch 2024-04-27T01:27:01.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:00 smithi162 ceph-mon[37680]: pgmap v1183: 176 pgs: 12 unknown, 164 active+clean; 583 KiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:01.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:00 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]: dispatch 2024-04-27T01:27:01.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:00 smithi181 ceph-mon[30237]: pgmap v1183: 176 pgs: 12 unknown, 164 active+clean; 583 KiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:01.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:00 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]: dispatch 2024-04-27T01:27:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:01.788+0000 7f167b7fe640 1 -- 172.21.15.162:0/1809297007 <== mon.0 v1:172.21.15.162:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]=0 enabled application 'rados' on pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' v762) ==== 213+0+0 (unknown 687248791 0 0) 0x7f167405f140 con 0x7f16841065a0 2024-04-27T01:27:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:enabled application 'rados' on pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' 2024-04-27T01:27:01.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:01.791+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 >> v1:172.21.15.162:6800/624703537 conn(0x7f1670071100 legacy=0x7f16700735c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:01.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:01.791+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 >> v1:172.21.15.162:6789/0 conn(0x7f16841065a0 legacy=0x7f1684102a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:01.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:01.791+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 shutdown_connections 2024-04-27T01:27:01.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:01.791+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 >> 172.21.15.162:0/1809297007 conn(0x7f16840fbfb0 msgr2=0x7f16840feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:27:01.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:01.791+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 shutdown_connections 2024-04-27T01:27:01.776 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:01.791+0000 7f1688aa4640 1 -- 172.21.15.162:0/1809297007 wait complete. 2024-04-27T01:27:01.790 INFO:tasks.workunit.client.0.smithi162.stderr:++ seq 1 10 2024-04-27T01:27:01.820 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:27:01.821 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put obj1 /etc/passwd 2024-04-27T01:27:01.857 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:27:01.857 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put obj2 /etc/passwd 2024-04-27T01:27:01.893 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:27:01.893 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put obj3 /etc/passwd 2024-04-27T01:27:01.926 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:27:01.947 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put obj4 /etc/passwd 2024-04-27T01:27:01.959 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:27:01.960 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put obj5 /etc/passwd 2024-04-27T01:27:01.995 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:27:01.995 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put obj6 /etc/passwd 2024-04-27T01:27:02.028 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:27:02.029 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put obj7 /etc/passwd 2024-04-27T01:27:02.061 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:27:02.061 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put obj8 /etc/passwd 2024-04-27T01:27:02.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]': finished 2024-04-27T01:27:02.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:01 smithi162 ceph-mon[37680]: osdmap e761: 8 total, 8 up, 8 in 2024-04-27T01:27:02.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:01 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]: dispatch 2024-04-27T01:27:02.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]': finished 2024-04-27T01:27:02.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:01 smithi162 ceph-mon[26714]: osdmap e761: 8 total, 8 up, 8 in 2024-04-27T01:27:02.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:01 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]: dispatch 2024-04-27T01:27:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:27:02.095 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put obj9 /etc/passwd 2024-04-27T01:27:02.127 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:27:02.127 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put obj10 /etc/passwd 2024-04-27T01:27:02.159 INFO:tasks.workunit.client.0.smithi162.stderr:+ sleep 30 2024-04-27T01:27:02.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]': finished 2024-04-27T01:27:02.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:01 smithi181 ceph-mon[30237]: osdmap e761: 8 total, 8 up, 8 in 2024-04-27T01:27:02.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:01 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]: dispatch 2024-04-27T01:27:03.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:02 smithi162 ceph-mon[26714]: pgmap v1186: 176 pgs: 12 unknown, 164 active+clean; 583 KiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:27:03.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:02 smithi162 ceph-mon[26714]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:27:03.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:02 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]': finished 2024-04-27T01:27:03.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:02 smithi162 ceph-mon[26714]: osdmap e762: 8 total, 8 up, 8 in 2024-04-27T01:27:03.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:02 smithi162 ceph-mon[37680]: pgmap v1186: 176 pgs: 12 unknown, 164 active+clean; 583 KiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:27:03.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:02 smithi162 ceph-mon[37680]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:27:03.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:02 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]': finished 2024-04-27T01:27:03.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:02 smithi162 ceph-mon[37680]: osdmap e762: 8 total, 8 up, 8 in 2024-04-27T01:27:03.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:27:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:27:03.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:02 smithi181 ceph-mon[30237]: pgmap v1186: 176 pgs: 12 unknown, 164 active+clean; 583 KiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:27:03.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:02 smithi181 ceph-mon[30237]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-27T01:27:03.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:02 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1809297007' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "app": "rados"}]': finished 2024-04-27T01:27:03.158 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:02 smithi181 ceph-mon[30237]: osdmap e762: 8 total, 8 up, 8 in 2024-04-27T01:27:04.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:04.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:04.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:05.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:04 smithi162 ceph-mon[37680]: pgmap v1188: 176 pgs: 176 active+clean; 699 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 KiB/s wr, 3 op/s 2024-04-27T01:27:05.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:04 smithi162 ceph-mon[37680]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' is full (reached quota's max_objects: 10) 2024-04-27T01:27:05.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:04 smithi162 ceph-mon[37680]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:27:05.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:04 smithi162 ceph-mon[37680]: osdmap e763: 8 total, 8 up, 8 in 2024-04-27T01:27:05.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:04 smithi162 ceph-mon[26714]: pgmap v1188: 176 pgs: 176 active+clean; 699 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 KiB/s wr, 3 op/s 2024-04-27T01:27:05.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:04 smithi162 ceph-mon[26714]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' is full (reached quota's max_objects: 10) 2024-04-27T01:27:05.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:04 smithi162 ceph-mon[26714]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:27:05.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:04 smithi162 ceph-mon[26714]: osdmap e763: 8 total, 8 up, 8 in 2024-04-27T01:27:05.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:04 smithi181 ceph-mon[30237]: pgmap v1188: 176 pgs: 176 active+clean; 699 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 26 KiB/s wr, 3 op/s 2024-04-27T01:27:05.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:04 smithi181 ceph-mon[30237]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' is full (reached quota's max_objects: 10) 2024-04-27T01:27:05.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:04 smithi181 ceph-mon[30237]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:27:05.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:04 smithi181 ceph-mon[30237]: osdmap e763: 8 total, 8 up, 8 in 2024-04-27T01:27:07.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:06 smithi181 ceph-mon[30237]: pgmap v1190: 176 pgs: 176 active+clean; 699 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 906 B/s rd, 21 KiB/s wr, 2 op/s 2024-04-27T01:27:07.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:06 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:07.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:06 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:07.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:06 smithi162 ceph-mon[37680]: pgmap v1190: 176 pgs: 176 active+clean; 699 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 906 B/s rd, 21 KiB/s wr, 2 op/s 2024-04-27T01:27:07.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:06 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:07.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:06 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:07.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:06 smithi162 ceph-mon[26714]: pgmap v1190: 176 pgs: 176 active+clean; 699 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 906 B/s rd, 21 KiB/s wr, 2 op/s 2024-04-27T01:27:07.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:06 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:07.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:06 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:09.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:08 smithi181 ceph-mon[30237]: pgmap v1191: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 KiB/s wr, 3 op/s 2024-04-27T01:27:09.086 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:08 smithi162 ceph-mon[37680]: pgmap v1191: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 KiB/s wr, 3 op/s 2024-04-27T01:27:09.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:08 smithi162 ceph-mon[26714]: pgmap v1191: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 KiB/s wr, 3 op/s 2024-04-27T01:27:10.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:27:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:27:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:27:11.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:10 smithi181 ceph-mon[30237]: pgmap v1192: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-04-27T01:27:11.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:10 smithi162 ceph-mon[26714]: pgmap v1192: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-04-27T01:27:11.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:10 smithi162 ceph-mon[37680]: pgmap v1192: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-04-27T01:27:12.910 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:12 smithi181 ceph-mon[30237]: pgmap v1193: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-04-27T01:27:13.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:12 smithi162 ceph-mon[37680]: pgmap v1193: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-04-27T01:27:13.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:12 smithi162 ceph-mon[26714]: pgmap v1193: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-04-27T01:27:13.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:27:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:27:13.985 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:13.985 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:14.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:15.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:14 smithi181 ceph-mon[30237]: pgmap v1194: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:27:15.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:14 smithi162 ceph-mon[37680]: pgmap v1194: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:27:15.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:14 smithi162 ceph-mon[26714]: pgmap v1194: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:27:17.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:16 smithi181 ceph-mon[30237]: pgmap v1195: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-04-27T01:27:17.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:16 smithi162 ceph-mon[37680]: pgmap v1195: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-04-27T01:27:17.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:16 smithi162 ceph-mon[26714]: pgmap v1195: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-04-27T01:27:19.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:18 smithi181 ceph-mon[30237]: pgmap v1196: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:19.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:18 smithi162 ceph-mon[37680]: pgmap v1196: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:19.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:18 smithi162 ceph-mon[26714]: pgmap v1196: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:20.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:27:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:27:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:27:21.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:20 smithi181 ceph-mon[30237]: pgmap v1197: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:21.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:20 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:21.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:20 smithi162 ceph-mon[37680]: pgmap v1197: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:21.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:20 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:21.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:20 smithi162 ceph-mon[26714]: pgmap v1197: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:21.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:20 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:23.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:27:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:27:23.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:22 smithi181 ceph-mon[30237]: pgmap v1198: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:23.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:22 smithi162 ceph-mon[37680]: pgmap v1198: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:23.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:22 smithi162 ceph-mon[26714]: pgmap v1198: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:24.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:24.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:24.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:25.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:24 smithi181 ceph-mon[30237]: pgmap v1199: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:25.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:24 smithi162 ceph-mon[37680]: pgmap v1199: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:25.086 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:24 smithi162 ceph-mon[26714]: pgmap v1199: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:27.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:26 smithi181 ceph-mon[30237]: pgmap v1200: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:27.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:26 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:27:27.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:26 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:27:27.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:26 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:27:27.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:26 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:27.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:26 smithi162 ceph-mon[26714]: pgmap v1200: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:27.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:26 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:27:27.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:26 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:27:27.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:26 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:27:27.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:26 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:27.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:26 smithi162 ceph-mon[37680]: pgmap v1200: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:27.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:26 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:27:27.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:26 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:27:27.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:26 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:27:27.085 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:26 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:29.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:28 smithi181 ceph-mon[30237]: pgmap v1201: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:29.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:28 smithi162 ceph-mon[37680]: pgmap v1201: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:29.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:28 smithi162 ceph-mon[26714]: pgmap v1201: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:30.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:27:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:27:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:27:31.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:30 smithi181 ceph-mon[30237]: pgmap v1202: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:31.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:30 smithi162 ceph-mon[26714]: pgmap v1202: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:31.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:30 smithi162 ceph-mon[37680]: pgmap v1202: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:32.161 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=145308 2024-04-27T01:27:32.161 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool set-quota 82d3730a-02a2-4166-9bba-a35fc501ddd8 max_objects 100 2024-04-27T01:27:32.161 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put onemore /etc/passwd 2024-04-27T01:27:32.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.270+0000 7f64fd44d640 1 -- 172.21.15.162:0/2681479309 >> v1:172.21.15.162:6790/0 conn(0x7f64f80ffeb0 legacy=0x7f64f8100290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:32.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.270+0000 7f64fd44d640 1 -- 172.21.15.162:0/2681479309 shutdown_connections 2024-04-27T01:27:32.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.270+0000 7f64fd44d640 1 -- 172.21.15.162:0/2681479309 >> 172.21.15.162:0/2681479309 conn(0x7f64f8076910 msgr2=0x7f64f80fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:27:32.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.270+0000 7f64fd44d640 1 -- 172.21.15.162:0/2681479309 shutdown_connections 2024-04-27T01:27:32.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.270+0000 7f64fd44d640 1 -- 172.21.15.162:0/2681479309 wait complete. 2024-04-27T01:27:32.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.271+0000 7f64fd44d640 1 Processor -- start 2024-04-27T01:27:32.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.271+0000 7f64fd44d640 1 -- start start 2024-04-27T01:27:32.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.271+0000 7f64fd44d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f64f81a67f0 con 0x7f64f8108b30 2024-04-27T01:27:32.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.271+0000 7f64fd44d640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f64f81a79d0 con 0x7f64f80ffeb0 2024-04-27T01:27:32.259 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.271+0000 7f64fd44d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f64f81a8bb0 con 0x7f64f810c6a0 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.271+0000 7f64f67fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f64f8108b30 0x7f64f81a19a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:34284/0 (socket says 172.21.15.162:34284) 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.271+0000 7f64f67fc640 1 -- 172.21.15.162:0/4084900950 learned_addr learned my addr 172.21.15.162:0/4084900950 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.271+0000 7f64f77fe640 1 --1- 172.21.15.162:0/4084900950 >> v1:172.21.15.162:6790/0 conn(0x7f64f810c6a0 0x7f64f81a50d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:52202/0 (socket says 172.21.15.162:52202) 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3346384593 0 0) 0x7f64f81a67f0 con 0x7f64f8108b30 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64e0003520 con 0x7f64f8108b30 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 471799703 0 0) 0x7f64f81a8bb0 con 0x7f64f810c6a0 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64f81a67f0 con 0x7f64f810c6a0 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 624499135 0 0) 0x7f64f81a79d0 con 0x7f64f80ffeb0 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64f81a8bb0 con 0x7f64f80ffeb0 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3792630170 0 0) 0x7f64e0003520 con 0x7f64f8108b30 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64f81a79d0 con 0x7f64f8108b30 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f64ec001930 con 0x7f64f8108b30 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1434307744 0 0) 0x7f64f81a67f0 con 0x7f64f810c6a0 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64e0003520 con 0x7f64f810c6a0 2024-04-27T01:27:32.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3209708986 0 0) 0x7f64f81a8bb0 con 0x7f64f80ffeb0 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64f81a67f0 con 0x7f64f80ffeb0 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f64f0001b70 con 0x7f64f810c6a0 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f64e8004450 con 0x7f64f80ffeb0 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1310310645 0 0) 0x7f64f81a79d0 con 0x7f64f8108b30 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 >> v1:172.21.15.162:6790/0 conn(0x7f64f810c6a0 legacy=0x7f64f81a50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 >> v1:172.21.15.181:6789/0 conn(0x7f64f80ffeb0 legacy=0x7f64f8104be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f81a9d90 con 0x7f64f8108b30 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f64f81a8de0 con 0x7f64f8108b30 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f64f81a92b0 con 0x7f64f8108b30 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f64ec003aa0 con 0x7f64f8108b30 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.272+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f64ec005c30 con 0x7f64f8108b30 2024-04-27T01:27:32.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.273+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64e4005080 con 0x7f64f8108b30 2024-04-27T01:27:32.262 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.274+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f64ec003610 con 0x7f64f8108b30 2024-04-27T01:27:32.262 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.274+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(763..763 src has 1..763) ==== 7754+0+0 (unknown 2317064966 0 0) 0x7f64ec089c20 con 0x7f64f8108b30 2024-04-27T01:27:32.262 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.275+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=764}) -- 0x7f64f81a79d0 con 0x7f64f8108b30 2024-04-27T01:27:32.266 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.279+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f64ec056d50 con 0x7f64f8108b30 2024-04-27T01:27:32.442 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.456+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"} v 0) -- 0x7f64e4005370 con 0x7f64f8108b30 2024-04-27T01:27:32.696 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.710+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v764) ==== 225+0+0 (unknown 184772036 0 0) 0x7f64ec05aa00 con 0x7f64f8108b30 2024-04-27T01:27:32.707 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.711+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 11 ==== osd_map(764..764 src has 1..764) ==== 628+0+0 (unknown 2782431217 0 0) 0x7f64ec087a00 con 0x7f64f8108b30 2024-04-27T01:27:32.707 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.711+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=765}) -- 0x7f64f81a67f0 con 0x7f64f8108b30 2024-04-27T01:27:32.809 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:32.823+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"} v 0) -- 0x7f64e4001090 con 0x7f64f8108b30 2024-04-27T01:27:33.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:27:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:27:33.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:32 smithi181 ceph-mon[30237]: pgmap v1203: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:33.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:32 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]: dispatch 2024-04-27T01:27:33.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:32 smithi162 ceph-mon[37680]: pgmap v1203: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:33.087 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:32 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]: dispatch 2024-04-27T01:27:33.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:32 smithi162 ceph-mon[26714]: pgmap v1203: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:33.087 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:32 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]: dispatch 2024-04-27T01:27:33.711 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:33.724+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v765) ==== 225+0+0 (unknown 601905596 0 0) 0x7f64ec05f2c0 con 0x7f64f8108b30 2024-04-27T01:27:33.711 INFO:tasks.workunit.client.0.smithi162.stderr:set-quota max_objects = 100 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 2024-04-27T01:27:33.711 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:33.725+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 <== mon.0 v1:172.21.15.162:6789/0 13 ==== osd_map(765..765 src has 1..765) ==== 628+0+0 (unknown 1594199924 0 0) 0x7f64ec052690 con 0x7f64f8108b30 2024-04-27T01:27:33.711 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:33.725+0000 7f64dffff640 1 -- 172.21.15.162:0/4084900950 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=766}) -- 0x7f64e0003520 con 0x7f64f8108b30 2024-04-27T01:27:33.713 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:33.727+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 >> v1:172.21.15.162:6800/624703537 conn(0x7f64e006ee50 legacy=0x7f64e0071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:33.713 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:33.727+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 >> v1:172.21.15.162:6789/0 conn(0x7f64f8108b30 legacy=0x7f64f81a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:33.714 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:33.728+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 shutdown_connections 2024-04-27T01:27:33.714 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:33.728+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 >> 172.21.15.162:0/4084900950 conn(0x7f64f8076910 msgr2=0x7f64f8113ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:27:33.714 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:33.728+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 shutdown_connections 2024-04-27T01:27:33.714 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:33.728+0000 7f64fd44d640 1 -- 172.21.15.162:0/4084900950 wait complete. 2024-04-27T01:27:33.728 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 145308 2024-04-27T01:27:34.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]': finished 2024-04-27T01:27:34.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:33 smithi181 ceph-mon[30237]: osdmap e764: 8 total, 8 up, 8 in 2024-04-27T01:27:34.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:33 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]: dispatch 2024-04-27T01:27:34.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:34.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]': finished 2024-04-27T01:27:34.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:33 smithi162 ceph-mon[37680]: osdmap e764: 8 total, 8 up, 8 in 2024-04-27T01:27:34.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:33 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]: dispatch 2024-04-27T01:27:34.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:34.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]': finished 2024-04-27T01:27:34.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:33 smithi162 ceph-mon[26714]: osdmap e764: 8 total, 8 up, 8 in 2024-04-27T01:27:34.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:33 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]: dispatch 2024-04-27T01:27:34.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:34.350 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 0 -ne 0 ']' 2024-04-27T01:27:34.350 INFO:tasks.workunit.client.0.smithi162.stderr:+ true 2024-04-27T01:27:34.350 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put twomore /etc/passwd 2024-04-27T01:27:34.387 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool set-quota 82d3730a-02a2-4166-9bba-a35fc501ddd8 max_bytes 100 2024-04-27T01:27:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.489+0000 7f2b313c9640 1 -- 172.21.15.162:0/2152777655 >> v1:172.21.15.162:6790/0 conn(0x7f2b2c102830 legacy=0x7f2b2c102c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.489+0000 7f2b313c9640 1 -- 172.21.15.162:0/2152777655 shutdown_connections 2024-04-27T01:27:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.489+0000 7f2b313c9640 1 -- 172.21.15.162:0/2152777655 >> 172.21.15.162:0/2152777655 conn(0x7f2b2c0fbfb0 msgr2=0x7f2b2c0fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:27:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.489+0000 7f2b313c9640 1 -- 172.21.15.162:0/2152777655 shutdown_connections 2024-04-27T01:27:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.489+0000 7f2b313c9640 1 -- 172.21.15.162:0/2152777655 wait complete. 2024-04-27T01:27:34.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.489+0000 7f2b313c9640 1 Processor -- start 2024-04-27T01:27:34.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.489+0000 7f2b313c9640 1 -- start start 2024-04-27T01:27:34.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b313c9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b2c1a88c0 con 0x7f2b2c10a630 2024-04-27T01:27:34.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b313c9640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b2c1a9b00 con 0x7f2b2c10e140 2024-04-27T01:27:34.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b313c9640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f2b2c1aad40 con 0x7f2b2c102830 2024-04-27T01:27:34.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b2affd640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f2b2c102830 0x7f2b2c109e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:52252/0 (socket says 172.21.15.162:52252) 2024-04-27T01:27:34.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b2affd640 1 -- 172.21.15.162:0/2237825234 learned_addr learned my addr 172.21.15.162:0/2237825234 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:27:34.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b2a7fc640 1 --1- 172.21.15.162:0/2237825234 >> v1:172.21.15.162:6789/0 conn(0x7f2b2c10a630 0x7f2b2c1a3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:34342/0 (socket says 172.21.15.162:34342) 2024-04-27T01:27:34.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223897455 0 0) 0x7f2b2c1aad40 con 0x7f2b2c102830 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b14003160 con 0x7f2b2c102830 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2661527583 0 0) 0x7f2b2c1a88c0 con 0x7f2b2c10a630 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b2c1aad40 con 0x7f2b2c10a630 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4217427652 0 0) 0x7f2b14003160 con 0x7f2b2c102830 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b2c1a88c0 con 0x7f2b2c102830 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2b1c0045f0 con 0x7f2b2c102830 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2489965608 0 0) 0x7f2b2c1aad40 con 0x7f2b2c10a630 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b14003160 con 0x7f2b2c10a630 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2b200018d0 con 0x7f2b2c10a630 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4212636877 0 0) 0x7f2b2c1a9b00 con 0x7f2b2c10e140 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b2c1aad40 con 0x7f2b2c10e140 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 266744728 0 0) 0x7f2b2c1a88c0 con 0x7f2b2c102830 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 >> v1:172.21.15.181:6789/0 conn(0x7f2b2c10e140 legacy=0x7f2b2c1a6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:34.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 >> v1:172.21.15.162:6789/0 conn(0x7f2b2c10a630 legacy=0x7f2b2c1a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:34.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2c1abf80 con 0x7f2b2c102830 2024-04-27T01:27:34.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.490+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b2c1a8af0 con 0x7f2b2c102830 2024-04-27T01:27:34.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.491+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b2c1a8fc0 con 0x7f2b2c102830 2024-04-27T01:27:34.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.491+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f2b1c001bd0 con 0x7f2b2c102830 2024-04-27T01:27:34.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.491+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2b1c004d90 con 0x7f2b2c102830 2024-04-27T01:27:34.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.492+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b18005080 con 0x7f2b2c102830 2024-04-27T01:27:34.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.492+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f2b1c004fd0 con 0x7f2b2c102830 2024-04-27T01:27:34.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.493+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(766..766 src has 1..766) ==== 7754+0+0 (unknown 444636883 0 0) 0x7f2b1c089c70 con 0x7f2b2c102830 2024-04-27T01:27:34.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.497+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f2b1c056ee0 con 0x7f2b2c102830 2024-04-27T01:27:34.661 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:34.675+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"} v 0) -- 0x7f2b18005370 con 0x7f2b2c102830 2024-04-27T01:27:35.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:34 smithi181 ceph-mon[30237]: pgmap v1205: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:27:35.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]': finished 2024-04-27T01:27:35.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:34 smithi181 ceph-mon[30237]: osdmap e765: 8 total, 8 up, 8 in 2024-04-27T01:27:35.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:34 smithi181 ceph-mon[30237]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' no longer out of quota; removing NO_QUOTA flag 2024-04-27T01:27:35.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:34 smithi181 ceph-mon[30237]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:27:35.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:34 smithi181 ceph-mon[30237]: osdmap e766: 8 total, 8 up, 8 in 2024-04-27T01:27:35.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:34 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2237825234' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:35.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:34 smithi181 ceph-mon[30237]: from='client.57197 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:35.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[37680]: pgmap v1205: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:27:35.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]': finished 2024-04-27T01:27:35.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[37680]: osdmap e765: 8 total, 8 up, 8 in 2024-04-27T01:27:35.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[37680]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' no longer out of quota; removing NO_QUOTA flag 2024-04-27T01:27:35.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[37680]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:27:35.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[37680]: osdmap e766: 8 total, 8 up, 8 in 2024-04-27T01:27:35.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2237825234' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:35.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[37680]: from='client.57197 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:35.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[26714]: pgmap v1205: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:27:35.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4084900950' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "100"}]': finished 2024-04-27T01:27:35.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[26714]: osdmap e765: 8 total, 8 up, 8 in 2024-04-27T01:27:35.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[26714]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' no longer out of quota; removing NO_QUOTA flag 2024-04-27T01:27:35.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[26714]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:27:35.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[26714]: osdmap e766: 8 total, 8 up, 8 in 2024-04-27T01:27:35.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2237825234' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:35.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:34 smithi162 ceph-mon[26714]: from='client.57197 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:35.344 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:35.359+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v767) ==== 221+0+0 (unknown 3283361944 0 0) 0x7f2b1c05ab90 con 0x7f2b2c102830 2024-04-27T01:27:35.452 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:35.466+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"} v 0) -- 0x7f2b18000c40 con 0x7f2b2c102830 2024-04-27T01:27:36.349 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:36.364+0000 7f2b13fff640 1 -- 172.21.15.162:0/2237825234 <== mon.2 v1:172.21.15.162:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v768) ==== 221+0+0 (unknown 1973930107 0 0) 0x7f2b1c05f450 con 0x7f2b2c102830 2024-04-27T01:27:36.349 INFO:tasks.workunit.client.0.smithi162.stderr:set-quota max_bytes = 100 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 2024-04-27T01:27:36.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:36.367+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 >> v1:172.21.15.162:6800/624703537 conn(0x7f2b1406efe0 legacy=0x7f2b140714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:36.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:36.367+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 >> v1:172.21.15.162:6790/0 conn(0x7f2b2c102830 legacy=0x7f2b2c109e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:27:36.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:36.367+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 shutdown_connections 2024-04-27T01:27:36.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:36.367+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 >> 172.21.15.162:0/2237825234 conn(0x7f2b2c0fbfb0 msgr2=0x7f2b2c0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:27:36.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:36.367+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 shutdown_connections 2024-04-27T01:27:36.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:27:36.367+0000 7f2b313c9640 1 -- 172.21.15.162:0/2237825234 wait complete. 2024-04-27T01:27:36.363 INFO:tasks.workunit.client.0.smithi162.stderr:+ sleep 30 2024-04-27T01:27:36.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:36 smithi181 ceph-mon[30237]: from='client.57197 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]': finished 2024-04-27T01:27:36.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:36 smithi181 ceph-mon[30237]: osdmap e767: 8 total, 8 up, 8 in 2024-04-27T01:27:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:36 smithi181 ceph-mon[30237]: pgmap v1209: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:36 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2237825234' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:36 smithi181 ceph-mon[30237]: from='client.57197 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:36 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:36.757 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:36.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[26714]: from='client.57197 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]': finished 2024-04-27T01:27:36.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[26714]: osdmap e767: 8 total, 8 up, 8 in 2024-04-27T01:27:36.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[26714]: pgmap v1209: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:36.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2237825234' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:36.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[26714]: from='client.57197 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:36.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:36.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[37680]: from='client.57197 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]': finished 2024-04-27T01:27:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[37680]: osdmap e767: 8 total, 8 up, 8 in 2024-04-27T01:27:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[37680]: pgmap v1209: 176 pgs: 176 active+clean; 699 KiB data, 829 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2237825234' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[37680]: from='client.57197 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]: dispatch 2024-04-27T01:27:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:36.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:37.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:37 smithi181 ceph-mon[30237]: from='client.57197 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]': finished 2024-04-27T01:27:37.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:37 smithi181 ceph-mon[30237]: osdmap e768: 8 total, 8 up, 8 in 2024-04-27T01:27:37.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:37 smithi162 ceph-mon[37680]: from='client.57197 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]': finished 2024-04-27T01:27:37.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:37 smithi162 ceph-mon[37680]: osdmap e768: 8 total, 8 up, 8 in 2024-04-27T01:27:37.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:37 smithi162 ceph-mon[26714]: from='client.57197 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "100"}]': finished 2024-04-27T01:27:37.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:37 smithi162 ceph-mon[26714]: osdmap e768: 8 total, 8 up, 8 in 2024-04-27T01:27:38.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:38 smithi181 ceph-mon[30237]: pgmap v1211: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-04-27T01:27:38.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:38 smithi162 ceph-mon[37680]: pgmap v1211: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-04-27T01:27:38.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:38 smithi162 ceph-mon[26714]: pgmap v1211: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-04-27T01:27:39.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:39 smithi162 ceph-mon[37680]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' is full (reached quota's max_bytes: 100 B) 2024-04-27T01:27:39.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:39 smithi162 ceph-mon[37680]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:27:39.726 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:39 smithi162 ceph-mon[37680]: osdmap e769: 8 total, 8 up, 8 in 2024-04-27T01:27:39.726 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:39 smithi162 ceph-mon[26714]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' is full (reached quota's max_bytes: 100 B) 2024-04-27T01:27:39.726 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:39 smithi162 ceph-mon[26714]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:27:39.726 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:39 smithi162 ceph-mon[26714]: osdmap e769: 8 total, 8 up, 8 in 2024-04-27T01:27:39.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:39 smithi181 ceph-mon[30237]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' is full (reached quota's max_bytes: 100 B) 2024-04-27T01:27:39.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:39 smithi181 ceph-mon[30237]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:27:39.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:39 smithi181 ceph-mon[30237]: osdmap e769: 8 total, 8 up, 8 in 2024-04-27T01:27:40.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:27:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:27:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:27:40.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:40 smithi181 ceph-mon[30237]: pgmap v1213: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1011 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-04-27T01:27:40.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:40 smithi162 ceph-mon[37680]: pgmap v1213: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1011 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-04-27T01:27:40.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:40 smithi162 ceph-mon[26714]: pgmap v1213: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1011 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-04-27T01:27:42.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:42 smithi181 ceph-mon[30237]: pgmap v1214: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 844 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:27:42.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:42 smithi162 ceph-mon[37680]: pgmap v1214: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 844 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:27:42.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:42 smithi162 ceph-mon[26714]: pgmap v1214: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 844 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:27:43.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:27:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:27:43.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:43.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:43.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:44.735 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:44 smithi162 ceph-mon[26714]: pgmap v1215: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-04-27T01:27:44.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:44 smithi162 ceph-mon[37680]: pgmap v1215: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-04-27T01:27:44.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:44 smithi181 ceph-mon[30237]: pgmap v1215: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-04-27T01:27:46.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:46 smithi181 ceph-mon[30237]: pgmap v1216: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 1 op/s 2024-04-27T01:27:46.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:46 smithi162 ceph-mon[26714]: pgmap v1216: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 1 op/s 2024-04-27T01:27:46.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:46 smithi162 ceph-mon[37680]: pgmap v1216: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 1 op/s 2024-04-27T01:27:48.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:48 smithi181 ceph-mon[30237]: pgmap v1217: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:27:48.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:48 smithi162 ceph-mon[26714]: pgmap v1217: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:27:48.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:48 smithi162 ceph-mon[37680]: pgmap v1217: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:27:50.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:27:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:27:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:27:50.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:50 smithi181 ceph-mon[30237]: pgmap v1218: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-04-27T01:27:50.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:50 smithi162 ceph-mon[37680]: pgmap v1218: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-04-27T01:27:50.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:50 smithi162 ceph-mon[26714]: pgmap v1218: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s 2024-04-27T01:27:52.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:51 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:52.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:51 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:52.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:51 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:52.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:51 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:52.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:51 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:27:52.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:51 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:27:53.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:27:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:27:53.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:52 smithi181 ceph-mon[30237]: pgmap v1219: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:53.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:52 smithi162 ceph-mon[26714]: pgmap v1219: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:53.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:52 smithi162 ceph-mon[37680]: pgmap v1219: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:54.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:53 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:54.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:53 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:54.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:53 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:27:55.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:54 smithi181 ceph-mon[30237]: pgmap v1220: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:55.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:54 smithi162 ceph-mon[37680]: pgmap v1220: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:55.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:54 smithi162 ceph-mon[26714]: pgmap v1220: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:57.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:56 smithi181 ceph-mon[30237]: pgmap v1221: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:57.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:56 smithi162 ceph-mon[37680]: pgmap v1221: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:57.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:56 smithi162 ceph-mon[26714]: pgmap v1221: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:27:59.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:27:58 smithi181 ceph-mon[30237]: pgmap v1222: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:59.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:27:58 smithi162 ceph-mon[37680]: pgmap v1222: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:27:59.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:27:58 smithi162 ceph-mon[26714]: pgmap v1222: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:00.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:27:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:27:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:28:01.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:00 smithi181 ceph-mon[30237]: pgmap v1223: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:01.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:00 smithi162 ceph-mon[26714]: pgmap v1223: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:01.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:00 smithi162 ceph-mon[37680]: pgmap v1223: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:03.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:28:02 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:28:03.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:02 smithi181 ceph-mon[30237]: pgmap v1224: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:03.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:02 smithi162 ceph-mon[37680]: pgmap v1224: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:03.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:02 smithi162 ceph-mon[26714]: pgmap v1224: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:04.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:03 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:04.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:03 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:04.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:03 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:05.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:04 smithi181 ceph-mon[30237]: pgmap v1225: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:05.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:04 smithi162 ceph-mon[37680]: pgmap v1225: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:05.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:04 smithi162 ceph-mon[26714]: pgmap v1225: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:06.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:05 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:06.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:05 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:06.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:05 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:06.365 INFO:tasks.workunit.client.0.smithi162.stderr:+ pid=145395 2024-04-27T01:28:06.365 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool set-quota 82d3730a-02a2-4166-9bba-a35fc501ddd8 max_bytes 0 2024-04-27T01:28:06.365 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put two /etc/passwd 2024-04-27T01:28:06.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.474+0000 7f876fbb1640 1 -- 172.21.15.162:0/3999868501 >> v1:172.21.15.162:6790/0 conn(0x7f8768114880 legacy=0x7f8768116c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:06.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.474+0000 7f876fbb1640 1 -- 172.21.15.162:0/3999868501 shutdown_connections 2024-04-27T01:28:06.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.474+0000 7f876fbb1640 1 -- 172.21.15.162:0/3999868501 >> 172.21.15.162:0/3999868501 conn(0x7f87680fbfb0 msgr2=0x7f87680fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:28:06.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.474+0000 7f876fbb1640 1 -- 172.21.15.162:0/3999868501 shutdown_connections 2024-04-27T01:28:06.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.474+0000 7f876fbb1640 1 -- 172.21.15.162:0/3999868501 wait complete. 2024-04-27T01:28:06.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.475+0000 7f876fbb1640 1 Processor -- start 2024-04-27T01:28:06.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.475+0000 7f876fbb1640 1 -- start start 2024-04-27T01:28:06.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.475+0000 7f876fbb1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8768114450 con 0x7f8768100580 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.475+0000 7f876fbb1640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f8768114680 con 0x7f8768114880 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.475+0000 7f876fbb1640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f87681af250 con 0x7f8768106370 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.475+0000 7f876d94d640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f8768100580 0x7f8768111320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52264/0 (socket says 172.21.15.162:52264) 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.475+0000 7f876d94d640 1 -- 172.21.15.162:0/2510523536 learned_addr learned my addr 172.21.15.162:0/2510523536 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.475+0000 7f876d14c640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f8768106370 0x7f8768111a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:43368/0 (socket says 172.21.15.162:43368) 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229424747 0 0) 0x7f8768114450 con 0x7f8768100580 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8754003520 con 0x7f8768100580 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1174373496 0 0) 0x7f87681af250 con 0x7f8768106370 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8768114450 con 0x7f8768106370 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988497837 0 0) 0x7f8768114680 con 0x7f8768114880 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87681af250 con 0x7f8768114880 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2396961594 0 0) 0x7f8754003520 con 0x7f8768100580 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8768114680 con 0x7f8768100580 2024-04-27T01:28:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8764001a30 con 0x7f8768100580 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1626263662 0 0) 0x7f8768114450 con 0x7f8768106370 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8754003520 con 0x7f8768106370 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4071162681 0 0) 0x7f87681af250 con 0x7f8768114880 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8768114450 con 0x7f8768114880 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8758001830 con 0x7f8768106370 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8760001b80 con 0x7f8768114880 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3509817646 0 0) 0x7f8768114680 con 0x7f8768100580 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 >> v1:172.21.15.162:6790/0 conn(0x7f8768106370 legacy=0x7f8768111a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 >> v1:172.21.15.181:6789/0 conn(0x7f8768114880 legacy=0x7f87681abad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87681b04b0 con 0x7f8768100580 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f87681af480 con 0x7f8768100580 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f87681af950 con 0x7f8768100580 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f8764001ce0 con 0x7f8768100580 2024-04-27T01:28:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.476+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f8764005cb0 con 0x7f8768100580 2024-04-27T01:28:06.466 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.477+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f874c005080 con 0x7f8768100580 2024-04-27T01:28:06.466 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.478+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f87640038a0 con 0x7f8768100580 2024-04-27T01:28:06.466 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.478+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(769..769 src has 1..769) ==== 7754+0+0 (unknown 3336495420 0 0) 0x7f8764089b50 con 0x7f8768100580 2024-04-27T01:28:06.466 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.479+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=770}) -- 0x7f8768114680 con 0x7f8768100580 2024-04-27T01:28:06.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.483+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f8764005cb0 con 0x7f8768100580 2024-04-27T01:28:06.645 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.661+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"} v 0) -- 0x7f874c005370 con 0x7f8768100580 2024-04-27T01:28:06.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.747+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v770) ==== 217+0+0 (unknown 2832232844 0 0) 0x7f8764056cc0 con 0x7f8768100580 2024-04-27T01:28:06.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.747+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 11 ==== osd_map(770..770 src has 1..770) ==== 628+0+0 (unknown 1988306966 0 0) 0x7f8764087970 con 0x7f8768100580 2024-04-27T01:28:06.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.747+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=771}) -- 0x7f87681af250 con 0x7f8768100580 2024-04-27T01:28:06.842 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:06.858+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"} v 0) -- 0x7f874c000c40 con 0x7f8768100580 2024-04-27T01:28:07.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:06 smithi181 ceph-mon[30237]: pgmap v1226: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:07.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:06 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:28:07.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:06 smithi162 ceph-mon[37680]: pgmap v1226: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:07.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:06 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:28:07.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:06 smithi162 ceph-mon[26714]: pgmap v1226: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:07.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:06 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:28:07.737 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.752+0000 7f875effd640 1 -- 172.21.15.162:0/2510523536 <== mon.0 v1:172.21.15.162:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v771) ==== 217+0+0 (unknown 2039184450 0 0) 0x7f876405a970 con 0x7f8768100580 2024-04-27T01:28:07.737 INFO:tasks.workunit.client.0.smithi162.stderr:set-quota max_bytes = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 2024-04-27T01:28:07.741 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.756+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 >> v1:172.21.15.162:6800/624703537 conn(0x7f875406eeb0 legacy=0x7f8754071370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:07.741 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.756+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 >> v1:172.21.15.162:6789/0 conn(0x7f8768100580 legacy=0x7f8768111320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:07.742 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.756+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 shutdown_connections 2024-04-27T01:28:07.742 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.756+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 >> 172.21.15.162:0/2510523536 conn(0x7f87680fbfb0 msgr2=0x7f87680fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:28:07.742 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.756+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 shutdown_connections 2024-04-27T01:28:07.742 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.756+0000 7f876fbb1640 1 -- 172.21.15.162:0/2510523536 wait complete. 2024-04-27T01:28:07.753 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool set-quota 82d3730a-02a2-4166-9bba-a35fc501ddd8 max_objects 0 2024-04-27T01:28:07.839 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.852+0000 7f0d41822640 1 -- 172.21.15.162:0/3193346081 >> v1:172.21.15.162:6789/0 conn(0x7f0d3c10a480 legacy=0x7f0d3c10c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:07.839 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.852+0000 7f0d41822640 1 -- 172.21.15.162:0/3193346081 shutdown_connections 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.852+0000 7f0d41822640 1 -- 172.21.15.162:0/3193346081 >> 172.21.15.162:0/3193346081 conn(0x7f0d3c0fd0c0 msgr2=0x7f0d3c0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.852+0000 7f0d41822640 1 -- 172.21.15.162:0/3193346081 shutdown_connections 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.852+0000 7f0d41822640 1 -- 172.21.15.162:0/3193346081 wait complete. 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d41822640 1 Processor -- start 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d41822640 1 -- start start 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d41822640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0d3c1a86e0 con 0x7f0d3c10a480 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d41822640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0d3c1a98c0 con 0x7f0d3c10e050 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d41822640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f0d3c1aab00 con 0x7f0d3c107670 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d33fff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f0d3c10a480 0x7f0d3c1a3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52296/0 (socket says 172.21.15.162:52296) 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d3affd640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f0d3c107670 0x7f0d3c1067f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:43384/0 (socket says 172.21.15.162:43384) 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d33fff640 1 -- 172.21.15.162:0/2464099423 learned_addr learned my addr 172.21.15.162:0/2464099423 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2746402894 0 0) 0x7f0d3c1a86e0 con 0x7f0d3c10a480 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d28003160 con 0x7f0d3c10a480 2024-04-27T01:28:07.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755558641 0 0) 0x7f0d3c1aab00 con 0x7f0d3c107670 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.853+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d3c1a86e0 con 0x7f0d3c107670 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1057691751 0 0) 0x7f0d3c1a98c0 con 0x7f0d3c10e050 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d3c1aab00 con 0x7f0d3c10e050 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2873428490 0 0) 0x7f0d28003160 con 0x7f0d3c10a480 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d3c1a98c0 con 0x7f0d3c10a480 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3023172328 0 0) 0x7f0d3c1a86e0 con 0x7f0d3c107670 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d28003160 con 0x7f0d3c107670 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f0d24004480 con 0x7f0d3c10a480 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f0d2c001890 con 0x7f0d3c107670 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 766831487 0 0) 0x7f0d3c1a98c0 con 0x7f0d3c10a480 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 >> v1:172.21.15.162:6790/0 conn(0x7f0d3c107670 legacy=0x7f0d3c1067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 >> v1:172.21.15.181:6789/0 conn(0x7f0d3c10e050 legacy=0x7f0d3c1a6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c1abd40 con 0x7f0d3c10a480 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0d3c1a8910 con 0x7f0d3c10a480 2024-04-27T01:28:07.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0d3c1a8de0 con 0x7f0d3c10a480 2024-04-27T01:28:07.842 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f0d24001b70 con 0x7f0d3c10a480 2024-04-27T01:28:07.842 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.854+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f0d24004de0 con 0x7f0d3c10a480 2024-04-27T01:28:07.842 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.855+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d1c005080 con 0x7f0d3c10a480 2024-04-27T01:28:07.842 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.855+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f0d24005060 con 0x7f0d3c10a480 2024-04-27T01:28:07.842 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.856+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(771..771 src has 1..771) ==== 7754+0+0 (unknown 834502868 0 0) 0x7f0d24089c50 con 0x7f0d3c10a480 2024-04-27T01:28:07.843 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.856+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=772}) -- 0x7f0d3c1a98c0 con 0x7f0d3c10a480 2024-04-27T01:28:07.845 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:07.860+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f0d24056dc0 con 0x7f0d3c10a480 2024-04-27T01:28:08.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:28:08.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:07 smithi181 ceph-mon[30237]: osdmap e770: 8 total, 8 up, 8 in 2024-04-27T01:28:08.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:07 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:28:08.023 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:08.039+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"} v 0) -- 0x7f0d1c005370 con 0x7f0d3c10a480 2024-04-27T01:28:08.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:28:08.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:07 smithi162 ceph-mon[37680]: osdmap e770: 8 total, 8 up, 8 in 2024-04-27T01:28:08.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:07 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:28:08.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:28:08.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:07 smithi162 ceph-mon[26714]: osdmap e770: 8 total, 8 up, 8 in 2024-04-27T01:28:08.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:07 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:28:08.743 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:08.758+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v772) ==== 221+0+0 (unknown 403104499 0 0) 0x7f0d2405aa70 con 0x7f0d3c10a480 2024-04-27T01:28:08.743 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:08.759+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 11 ==== osd_map(772..772 src has 1..772) ==== 628+0+0 (unknown 2782657709 0 0) 0x7f0d24087a70 con 0x7f0d3c10a480 2024-04-27T01:28:08.743 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:08.759+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=773}) -- 0x7f0d3c1a86e0 con 0x7f0d3c10a480 2024-04-27T01:28:08.854 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:08.869+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"} v 0) -- 0x7f0d1c000c40 con 0x7f0d3c10a480 2024-04-27T01:28:09.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:08 smithi181 ceph-mon[30237]: pgmap v1228: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:28:09.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:28:09.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:08 smithi181 ceph-mon[30237]: osdmap e771: 8 total, 8 up, 8 in 2024-04-27T01:28:09.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:08 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:28:09.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:08 smithi162 ceph-mon[37680]: pgmap v1228: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:28:09.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:28:09.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:08 smithi162 ceph-mon[37680]: osdmap e771: 8 total, 8 up, 8 in 2024-04-27T01:28:09.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:08 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:28:09.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:08 smithi162 ceph-mon[26714]: pgmap v1228: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:28:09.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2510523536' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:28:09.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:08 smithi162 ceph-mon[26714]: osdmap e771: 8 total, 8 up, 8 in 2024-04-27T01:28:09.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:08 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:28:09.344 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.359+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v773) ==== 221+0+0 (unknown 2555786002 0 0) 0x7f0d2405f330 con 0x7f0d3c10a480 2024-04-27T01:28:09.344 INFO:tasks.workunit.client.0.smithi162.stderr:set-quota max_objects = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 2024-04-27T01:28:09.345 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.360+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 <== mon.0 v1:172.21.15.162:6789/0 13 ==== osd_map(773..773 src has 1..773) ==== 628+0+0 (unknown 1558292791 0 0) 0x7f0d24087a70 con 0x7f0d3c10a480 2024-04-27T01:28:09.345 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.360+0000 7f0d40820640 1 -- 172.21.15.162:0/2464099423 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=774}) -- 0x7f0d2807bea0 con 0x7f0d3c10a480 2024-04-27T01:28:09.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.363+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 >> v1:172.21.15.162:6800/624703537 conn(0x7f0d2806ee60 legacy=0x7f0d28071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:09.348 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.363+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 >> v1:172.21.15.162:6789/0 conn(0x7f0d3c10a480 legacy=0x7f0d3c1a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:09.349 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.365+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 shutdown_connections 2024-04-27T01:28:09.349 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.365+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 >> 172.21.15.162:0/2464099423 conn(0x7f0d3c0fd0c0 msgr2=0x7f0d3c10e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:28:09.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.365+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 shutdown_connections 2024-04-27T01:28:09.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.365+0000 7f0d41822640 1 -- 172.21.15.162:0/2464099423 wait complete. 2024-04-27T01:28:09.363 INFO:tasks.workunit.client.0.smithi162.stderr:+ wait 145395 2024-04-27T01:28:09.363 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' 0 -ne 0 ']' 2024-04-27T01:28:09.363 INFO:tasks.workunit.client.0.smithi162.stderr:+ true 2024-04-27T01:28:09.363 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put three /etc/passwd 2024-04-27T01:28:09.397 INFO:tasks.workunit.client.0.smithi162.stderr:++ uuidgen 2024-04-27T01:28:09.398 INFO:tasks.workunit.client.0.smithi162.stderr:+ pp=0d36d9ee-268f-4098-865d-4f6f3764d7d6 2024-04-27T01:28:09.398 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool create 0d36d9ee-268f-4098-865d-4f6f3764d7d6 12 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.495+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/1409583712 >> v1:172.21.15.162:6790/0 conn(0x7fdfe010cd90 legacy=0x7fdfe010f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.495+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/1409583712 shutdown_connections 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.495+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/1409583712 >> 172.21.15.162:0/1409583712 conn(0x7fdfe0075ed0 msgr2=0x7fdfe00762e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.495+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/1409583712 shutdown_connections 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.495+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/1409583712 wait complete. 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.496+0000 7fdfe6f9f640 1 Processor -- start 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.496+0000 7fdfe6f9f640 1 -- start start 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.496+0000 7fdfe6f9f640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdfe01a6400 con 0x7fdfe010cd90 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.496+0000 7fdfe6f9f640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdfe01a7640 con 0x7fdfe0109220 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.496+0000 7fdfe6f9f640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fdfe01a8880 con 0x7fdfe01006b0 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.496+0000 7fdfe5f9d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fdfe01006b0 0x7fdfe0069130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:43420/0 (socket says 172.21.15.162:43420) 2024-04-27T01:28:09.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.496+0000 7fdfe5f9d640 1 -- 172.21.15.162:0/715109902 learned_addr learned my addr 172.21.15.162:0/715109902 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1287002279 0 0) 0x7fdfe01a8880 con 0x7fdfe01006b0 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfcc003520 con 0x7fdfe01006b0 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 490672756 0 0) 0x7fdfe01a6400 con 0x7fdfe010cd90 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfe01a8880 con 0x7fdfe010cd90 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1333977083 0 0) 0x7fdfe01a7640 con 0x7fdfe0109220 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfe01a6400 con 0x7fdfe0109220 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1717823966 0 0) 0x7fdfcc003520 con 0x7fdfe01006b0 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfe01a7640 con 0x7fdfe01006b0 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fdfdc001a30 con 0x7fdfe01006b0 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 405154044 0 0) 0x7fdfe01a8880 con 0x7fdfe010cd90 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfcc003520 con 0x7fdfe010cd90 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fdfd8001bc0 con 0x7fdfe010cd90 2024-04-27T01:28:09.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2182642596 0 0) 0x7fdfe01a6400 con 0x7fdfe0109220 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdfe01a8880 con 0x7fdfe0109220 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fdfd00017b0 con 0x7fdfe0109220 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1375008094 0 0) 0x7fdfe01a7640 con 0x7fdfe01006b0 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 >> v1:172.21.15.181:6789/0 conn(0x7fdfe0109220 legacy=0x7fdfe01a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 >> v1:172.21.15.162:6789/0 conn(0x7fdfe010cd90 legacy=0x7fdfe01a4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe01a9ac0 con 0x7fdfe01006b0 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe01a6630 con 0x7fdfe01006b0 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdfe01a6b00 con 0x7fdfe01006b0 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fdfdc001ce0 con 0x7fdfe01006b0 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.497+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fdfdc005cb0 con 0x7fdfe01006b0 2024-04-27T01:28:09.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.498+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfc4005080 con 0x7fdfe01006b0 2024-04-27T01:28:09.486 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.499+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7fdfdc0038a0 con 0x7fdfe01006b0 2024-04-27T01:28:09.486 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.499+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(773..773 src has 1..773) ==== 7754+0+0 (unknown 770987037 0 0) 0x7fdfdc089d20 con 0x7fdfe01006b0 2024-04-27T01:28:09.488 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.504+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fdfdc056e50 con 0x7fdfe01006b0 2024-04-27T01:28:09.668 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:09.684+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12} v 0) -- 0x7fdfc4005370 con 0x7fdfe01006b0 2024-04-27T01:28:10.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:28:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:28:10.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:28:10.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[37680]: osdmap e772: 8 total, 8 up, 8 in 2024-04-27T01:28:10.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:28:10.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[37680]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' no longer out of quota; removing NO_QUOTA flag 2024-04-27T01:28:10.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[37680]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:28:10.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:28:10.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[37680]: osdmap e773: 8 total, 8 up, 8 in 2024-04-27T01:28:10.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:28:10.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[26714]: osdmap e772: 8 total, 8 up, 8 in 2024-04-27T01:28:10.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:28:10.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[26714]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' no longer out of quota; removing NO_QUOTA flag 2024-04-27T01:28:10.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[26714]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:28:10.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:28:10.085 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:09 smithi162 ceph-mon[26714]: osdmap e773: 8 total, 8 up, 8 in 2024-04-27T01:28:10.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:28:10.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:09 smithi181 ceph-mon[30237]: osdmap e772: 8 total, 8 up, 8 in 2024-04-27T01:28:10.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:28:10.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:09 smithi181 ceph-mon[30237]: pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' no longer out of quota; removing NO_QUOTA flag 2024-04-27T01:28:10.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:09 smithi181 ceph-mon[30237]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:28:10.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:09 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/2464099423' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:28:10.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:09 smithi181 ceph-mon[30237]: osdmap e773: 8 total, 8 up, 8 in 2024-04-27T01:28:10.354 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.370+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]=0 pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' created v774) ==== 176+0+0 (unknown 2625046606 0 0) 0x7fdfdc05ab00 con 0x7fdfe01006b0 2024-04-27T01:28:10.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.479+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12} v 0) -- 0x7fdfc4001170 con 0x7fdfe01006b0 2024-04-27T01:28:10.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.480+0000 7fdfd6ffd640 1 -- 172.21.15.162:0/715109902 <== mon.2 v1:172.21.15.162:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]=0 pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' already exists v774) ==== 183+0+0 (unknown 987785167 0 0) 0x7fdfc4001170 con 0x7fdfe01006b0 2024-04-27T01:28:10.464 INFO:tasks.workunit.client.0.smithi162.stderr:pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' already exists 2024-04-27T01:28:10.466 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.482+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 >> v1:172.21.15.162:6800/624703537 conn(0x7fdfcc06ef30 legacy=0x7fdfcc0713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:10.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.482+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 >> v1:172.21.15.162:6790/0 conn(0x7fdfe01006b0 legacy=0x7fdfe0069130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:10.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.482+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 shutdown_connections 2024-04-27T01:28:10.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.482+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 >> 172.21.15.162:0/715109902 conn(0x7fdfe0075ed0 msgr2=0x7fdfe01102b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:28:10.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.482+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 shutdown_connections 2024-04-27T01:28:10.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.482+0000 7fdfe6f9f640 1 -- 172.21.15.162:0/715109902 wait complete. 2024-04-27T01:28:10.475 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool application enable 0d36d9ee-268f-4098-865d-4f6f3764d7d6 rados 2024-04-27T01:28:10.560 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.573+0000 7f650f874640 1 -- 172.21.15.162:0/3040390202 >> v1:172.21.15.162:6790/0 conn(0x7f6508109330 legacy=0x7f650810b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.573+0000 7f650f874640 1 -- 172.21.15.162:0/3040390202 shutdown_connections 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.573+0000 7f650f874640 1 -- 172.21.15.162:0/3040390202 >> 172.21.15.162:0/3040390202 conn(0x7f65080fbf80 msgr2=0x7f65080fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.573+0000 7f650f874640 1 -- 172.21.15.162:0/3040390202 shutdown_connections 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.573+0000 7f650f874640 1 -- 172.21.15.162:0/3040390202 wait complete. 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f650f874640 1 Processor -- start 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f650f874640 1 -- start start 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f650f874640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65080fe100 con 0x7f6508109330 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f650f874640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65080fe310 con 0x7f650810cf30 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f650f874640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f65081aaf10 con 0x7f6508106590 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f650ce0f640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6508109330 0x7f65080fcb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52382/0 (socket says 172.21.15.162:52382) 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f650d610640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f6508106590 0x7f65080fc400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:43438/0 (socket says 172.21.15.162:43438) 2024-04-27T01:28:10.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f650ce0f640 1 -- 172.21.15.162:0/3489186200 learned_addr learned my addr 172.21.15.162:0/3489186200 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3791763844 0 0) 0x7f65081aaf10 con 0x7f6508106590 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64f4003520 con 0x7f6508106590 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2039398426 0 0) 0x7f65080fe100 con 0x7f6508109330 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65081aaf10 con 0x7f6508109330 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.574+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2919644439 0 0) 0x7f65080fe310 con 0x7f650810cf30 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65080fe100 con 0x7f650810cf30 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1256756947 0 0) 0x7f64f4003520 con 0x7f6508106590 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65080fe310 con 0x7f6508106590 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f65040018b0 con 0x7f6508106590 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1945597365 0 0) 0x7f65081aaf10 con 0x7f6508109330 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64f4003520 con 0x7f6508109330 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f64f8001880 con 0x7f6508109330 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3610672670 0 0) 0x7f65080fe100 con 0x7f650810cf30 2024-04-27T01:28:10.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65081aaf10 con 0x7f650810cf30 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6500001c50 con 0x7f650810cf30 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 908772343 0 0) 0x7f65080fe310 con 0x7f6508106590 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 >> v1:172.21.15.181:6789/0 conn(0x7f650810cf30 legacy=0x7f65081a77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 >> v1:172.21.15.162:6789/0 conn(0x7f6508109330 legacy=0x7f65080fcb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65081ac130 con 0x7f6508106590 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65081ab140 con 0x7f6508106590 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f65081ab610 con 0x7f6508106590 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f6504003aa0 con 0x7f6508106590 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.575+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f6504005b90 con 0x7f6508106590 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.576+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64ec005080 con 0x7f6508106590 2024-04-27T01:28:10.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.576+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f6504003610 con 0x7f6508106590 2024-04-27T01:28:10.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.577+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(774..774 src has 1..774) ==== 8129+0+0 (unknown 860008314 0 0) 0x7f6504089af0 con 0x7f6508106590 2024-04-27T01:28:10.566 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.581+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f6504056aa0 con 0x7f6508106590 2024-04-27T01:28:10.744 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:10.760+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"} v 0) -- 0x7f64ec005370 con 0x7f6508106590 2024-04-27T01:28:11.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:10 smithi181 ceph-mon[30237]: pgmap v1232: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:11.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/715109902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:10 smithi181 ceph-mon[30237]: from='client.57233 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:10 smithi181 ceph-mon[30237]: from='client.57233 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]': finished 2024-04-27T01:28:11.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:10 smithi181 ceph-mon[30237]: osdmap e774: 8 total, 8 up, 8 in 2024-04-27T01:28:11.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/715109902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:10 smithi181 ceph-mon[30237]: from='client.57233 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:10 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3489186200' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:11.257 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:10 smithi181 ceph-mon[30237]: from='client.57239 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:11.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[37680]: pgmap v1232: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:11.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/715109902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[37680]: from='client.57233 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[37680]: from='client.57233 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]': finished 2024-04-27T01:28:11.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[37680]: osdmap e774: 8 total, 8 up, 8 in 2024-04-27T01:28:11.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/715109902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[37680]: from='client.57233 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3489186200' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:11.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[37680]: from='client.57239 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:11.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[26714]: pgmap v1232: 176 pgs: 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:11.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/715109902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[26714]: from='client.57233 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[26714]: from='client.57233 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]': finished 2024-04-27T01:28:11.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[26714]: osdmap e774: 8 total, 8 up, 8 in 2024-04-27T01:28:11.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/715109902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[26714]: from='client.57233 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pg_num": 12}]: dispatch 2024-04-27T01:28:11.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3489186200' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:11.335 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:10 smithi162 ceph-mon[26714]: from='client.57239 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:11.360 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:11.375+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]=0 enabled application 'rados' on pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' v775) ==== 213+0+0 (unknown 291025023 0 0) 0x7f650405a750 con 0x7f6508106590 2024-04-27T01:28:11.474 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:11.489+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"} v 0) -- 0x7f64ec001090 con 0x7f6508106590 2024-04-27T01:28:12.370 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.385+0000 7f64fe7fc640 1 -- 172.21.15.162:0/3489186200 <== mon.2 v1:172.21.15.162:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]=0 enabled application 'rados' on pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' v776) ==== 213+0+0 (unknown 1667150092 0 0) 0x7f650405f010 con 0x7f6508106590 2024-04-27T01:28:12.371 INFO:tasks.workunit.client.0.smithi162.stderr:enabled application 'rados' on pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' 2024-04-27T01:28:12.373 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.387+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 >> v1:172.21.15.162:6800/624703537 conn(0x7f64f406ef40 legacy=0x7f64f4071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:12.373 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.387+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 >> v1:172.21.15.162:6790/0 conn(0x7f6508106590 legacy=0x7f65080fc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:12.373 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.387+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 shutdown_connections 2024-04-27T01:28:12.373 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.387+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 >> 172.21.15.162:0/3489186200 conn(0x7f65080fbf80 msgr2=0x7f650810ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:28:12.373 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.388+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 shutdown_connections 2024-04-27T01:28:12.373 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.388+0000 7f650f874640 1 -- 172.21.15.162:0/3489186200 wait complete. 2024-04-27T01:28:12.384 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool set-quota 0d36d9ee-268f-4098-865d-4f6f3764d7d6 max_objects 10 2024-04-27T01:28:12.474 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.486+0000 7f2926dbf640 1 -- 172.21.15.162:0/54439733 >> v1:172.21.15.162:6790/0 conn(0x7f292010a5a0 legacy=0x7f292010c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:12.474 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.486+0000 7f2926dbf640 1 -- 172.21.15.162:0/54439733 shutdown_connections 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.486+0000 7f2926dbf640 1 -- 172.21.15.162:0/54439733 >> 172.21.15.162:0/54439733 conn(0x7f29200771a0 msgr2=0x7f29200fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.486+0000 7f2926dbf640 1 -- 172.21.15.162:0/54439733 shutdown_connections 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.486+0000 7f2926dbf640 1 -- 172.21.15.162:0/54439733 wait complete. 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.486+0000 7f2926dbf640 1 Processor -- start 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f2926dbf640 1 -- start start 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f2926dbf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f292010a250 con 0x7f29201027a0 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f2926dbf640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29201a7c10 con 0x7f292010a5a0 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f2926dbf640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f29201a8df0 con 0x7f292010e0b0 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f2924b5b640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f29201027a0 0x7f29201070d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52400/0 (socket says 172.21.15.162:52400) 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f2924b5b640 1 -- 172.21.15.162:0/4207484578 learned_addr learned my addr 172.21.15.162:0/4207484578 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f292535c640 1 --1- 172.21.15.162:0/4207484578 >> v1:172.21.15.162:6790/0 conn(0x7f292010e0b0 0x7f29201a54e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:43446/0 (socket says 172.21.15.162:43446) 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1272553156 0 0) 0x7f292010a250 con 0x7f29201027a0 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2908003520 con 0x7f29201027a0 2024-04-27T01:28:12.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3845752587 0 0) 0x7f29201a8df0 con 0x7f292010e0b0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f292010a250 con 0x7f292010e0b0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1648700734 0 0) 0x7f29201a7c10 con 0x7f292010a5a0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29201a8df0 con 0x7f292010a5a0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3892998279 0 0) 0x7f2908003520 con 0x7f29201027a0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29201a7c10 con 0x7f29201027a0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f29100018b0 con 0x7f29201027a0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3907782092 0 0) 0x7f292010a250 con 0x7f292010e0b0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2908003520 con 0x7f292010e0b0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2918001bb0 con 0x7f292010e0b0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 430141677 0 0) 0x7f29201a8df0 con 0x7f292010a5a0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f292010a250 con 0x7f292010a5a0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.487+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2914001430 con 0x7f292010a5a0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.488+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2212767145 0 0) 0x7f29201a7c10 con 0x7f29201027a0 2024-04-27T01:28:12.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.488+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 >> v1:172.21.15.162:6790/0 conn(0x7f292010e0b0 legacy=0x7f29201a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:12.477 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.488+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 >> v1:172.21.15.181:6789/0 conn(0x7f292010a5a0 legacy=0x7f29201077e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:12.477 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.488+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29201a9fd0 con 0x7f29201027a0 2024-04-27T01:28:12.477 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.488+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29201a9020 con 0x7f29201027a0 2024-04-27T01:28:12.477 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.488+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f2910001a50 con 0x7f29201027a0 2024-04-27T01:28:12.477 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.488+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f2910005b80 con 0x7f29201027a0 2024-04-27T01:28:12.477 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.488+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f29201a94f0 con 0x7f29201027a0 2024-04-27T01:28:12.477 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.489+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2920069130 con 0x7f29201027a0 2024-04-27T01:28:12.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.489+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f29100035f0 con 0x7f29201027a0 2024-04-27T01:28:12.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.490+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(776..776 src has 1..776) ==== 8142+0+0 (unknown 387521053 0 0) 0x7f2910089bf0 con 0x7f29201027a0 2024-04-27T01:28:12.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.494+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f2910056c00 con 0x7f29201027a0 2024-04-27T01:28:12.657 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:12.672+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"} v 0) -- 0x7f2920068400 con 0x7f29201027a0 2024-04-27T01:28:12.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:12 smithi181 ceph-mon[30237]: from='client.57239 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]': finished 2024-04-27T01:28:12.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:12 smithi181 ceph-mon[30237]: osdmap e775: 8 total, 8 up, 8 in 2024-04-27T01:28:12.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:12 smithi181 ceph-mon[30237]: pgmap v1235: 188 pgs: 12 unknown, 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:28:12.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:12 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3489186200' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:12.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:12 smithi181 ceph-mon[30237]: from='client.57239 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:12.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:12 smithi162 ceph-mon[37680]: from='client.57239 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]': finished 2024-04-27T01:28:12.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:12 smithi162 ceph-mon[37680]: osdmap e775: 8 total, 8 up, 8 in 2024-04-27T01:28:12.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:12 smithi162 ceph-mon[37680]: pgmap v1235: 188 pgs: 12 unknown, 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:28:12.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:12 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3489186200' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:12.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:12 smithi162 ceph-mon[37680]: from='client.57239 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:12.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:12 smithi162 ceph-mon[26714]: from='client.57239 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]': finished 2024-04-27T01:28:12.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:12 smithi162 ceph-mon[26714]: osdmap e775: 8 total, 8 up, 8 in 2024-04-27T01:28:12.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:12 smithi162 ceph-mon[26714]: pgmap v1235: 188 pgs: 12 unknown, 176 active+clean; 722 KiB data, 830 MiB used, 714 GiB / 715 GiB avail 2024-04-27T01:28:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:12 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3489186200' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:12.835 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:12 smithi162 ceph-mon[26714]: from='client.57239 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]: dispatch 2024-04-27T01:28:13.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:28:12 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:28:13.367 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:13.381+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 0d36d9ee-268f-4098-865d-4f6f3764d7d6 v777) ==== 223+0+0 (unknown 3214860160 0 0) 0x7f291005a8b0 con 0x7f29201027a0 2024-04-27T01:28:13.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:13.491+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"} v 0) -- 0x7f2920113e50 con 0x7f29201027a0 2024-04-27T01:28:13.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:13 smithi181 ceph-mon[30237]: from='client.57239 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]': finished 2024-04-27T01:28:13.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:13 smithi181 ceph-mon[30237]: osdmap e776: 8 total, 8 up, 8 in 2024-04-27T01:28:13.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:13 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:28:13.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:13 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:13.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:13 smithi162 ceph-mon[26714]: from='client.57239 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]': finished 2024-04-27T01:28:13.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:13 smithi162 ceph-mon[26714]: osdmap e776: 8 total, 8 up, 8 in 2024-04-27T01:28:13.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:13 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:28:13.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:13 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:13.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:13 smithi162 ceph-mon[37680]: from='client.57239 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "app": "rados"}]': finished 2024-04-27T01:28:13.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:13 smithi162 ceph-mon[37680]: osdmap e776: 8 total, 8 up, 8 in 2024-04-27T01:28:13.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:13 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:28:13.835 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:13 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:14.373 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:14.387+0000 7f291dffb640 1 -- 172.21.15.162:0/4207484578 <== mon.0 v1:172.21.15.162:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 0d36d9ee-268f-4098-865d-4f6f3764d7d6 v778) ==== 223+0+0 (unknown 1682976784 0 0) 0x7f291005f170 con 0x7f29201027a0 2024-04-27T01:28:14.373 INFO:tasks.workunit.client.0.smithi162.stderr:set-quota max_objects = 10 for pool 0d36d9ee-268f-4098-865d-4f6f3764d7d6 2024-04-27T01:28:14.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:14.391+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 >> v1:172.21.15.162:6800/624703537 conn(0x7f290806ee60 legacy=0x7f2908071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:14.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:14.391+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 >> v1:172.21.15.162:6789/0 conn(0x7f29201027a0 legacy=0x7f29201070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:28:14.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:14.391+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 shutdown_connections 2024-04-27T01:28:14.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:14.391+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 >> 172.21.15.162:0/4207484578 conn(0x7f29200771a0 msgr2=0x7f292010fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:28:14.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:14.392+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 shutdown_connections 2024-04-27T01:28:14.378 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:28:14.392+0000 7f2926dbf640 1 -- 172.21.15.162:0/4207484578 wait complete. 2024-04-27T01:28:14.390 INFO:tasks.workunit.client.0.smithi162.stderr:+ sleep 30 2024-04-27T01:28:14.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:28:14.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:14 smithi162 ceph-mon[37680]: osdmap e777: 8 total, 8 up, 8 in 2024-04-27T01:28:14.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:14 smithi162 ceph-mon[37680]: pgmap v1238: 188 pgs: 188 active+clean; 745 KiB data, 848 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-04-27T01:28:14.735 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:14 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:28:14.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:28:14.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:14 smithi162 ceph-mon[26714]: osdmap e777: 8 total, 8 up, 8 in 2024-04-27T01:28:14.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:14 smithi162 ceph-mon[26714]: pgmap v1238: 188 pgs: 188 active+clean; 745 KiB data, 848 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-04-27T01:28:14.736 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:14 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:28:14.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:28:14.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:14 smithi181 ceph-mon[30237]: osdmap e777: 8 total, 8 up, 8 in 2024-04-27T01:28:14.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:14 smithi181 ceph-mon[30237]: pgmap v1238: 188 pgs: 188 active+clean; 745 KiB data, 848 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-04-27T01:28:14.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:14 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]: dispatch 2024-04-27T01:28:15.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:28:15.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:15 smithi181 ceph-mon[30237]: osdmap e778: 8 total, 8 up, 8 in 2024-04-27T01:28:15.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:28:15.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:15 smithi162 ceph-mon[37680]: osdmap e778: 8 total, 8 up, 8 in 2024-04-27T01:28:15.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/4207484578' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "field": "max_objects", "val": "10"}]': finished 2024-04-27T01:28:15.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:15 smithi162 ceph-mon[26714]: osdmap e778: 8 total, 8 up, 8 in 2024-04-27T01:28:16.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:16 smithi181 ceph-mon[30237]: pgmap v1240: 188 pgs: 188 active+clean; 745 KiB data, 848 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.9 KiB/s wr, 1 op/s 2024-04-27T01:28:16.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:16 smithi162 ceph-mon[37680]: pgmap v1240: 188 pgs: 188 active+clean; 745 KiB data, 848 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.9 KiB/s wr, 1 op/s 2024-04-27T01:28:16.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:16 smithi162 ceph-mon[26714]: pgmap v1240: 188 pgs: 188 active+clean; 745 KiB data, 848 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.9 KiB/s wr, 1 op/s 2024-04-27T01:28:18.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:18 smithi181 ceph-mon[30237]: pgmap v1241: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:28:18.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:18 smithi162 ceph-mon[37680]: pgmap v1241: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:28:18.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:18 smithi162 ceph-mon[26714]: pgmap v1241: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-04-27T01:28:20.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:28:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:28:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:28:20.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:20 smithi181 ceph-mon[30237]: pgmap v1242: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 726 B/s rd, 0 op/s 2024-04-27T01:28:20.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:20 smithi162 ceph-mon[26714]: pgmap v1242: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 726 B/s rd, 0 op/s 2024-04-27T01:28:20.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:20 smithi162 ceph-mon[37680]: pgmap v1242: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 726 B/s rd, 0 op/s 2024-04-27T01:28:22.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:21 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:28:22.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:21 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:22.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:21 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:28:22.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:21 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:22.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:21 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:28:22.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:21 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:23.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:28:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:28:23.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:22 smithi181 ceph-mon[30237]: pgmap v1243: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-04-27T01:28:23.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:22 smithi162 ceph-mon[26714]: pgmap v1243: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-04-27T01:28:23.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:22 smithi162 ceph-mon[37680]: pgmap v1243: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 636 B/s rd, 0 op/s 2024-04-27T01:28:24.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:24.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:24.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:25.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:24 smithi181 ceph-mon[30237]: pgmap v1244: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:28:25.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:24 smithi162 ceph-mon[26714]: pgmap v1244: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:28:25.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:24 smithi162 ceph-mon[37680]: pgmap v1244: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:28:27.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:26 smithi181 ceph-mon[30237]: pgmap v1245: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-04-27T01:28:27.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:26 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:28:27.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:26 smithi162 ceph-mon[37680]: pgmap v1245: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-04-27T01:28:27.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:26 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:28:27.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:26 smithi162 ceph-mon[26714]: pgmap v1245: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-04-27T01:28:27.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:26 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:28:28.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:27 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:28:28.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:27 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:28:28.007 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:27 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:28:28.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:27 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:28:28.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:27 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:28:28.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:27 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:28:28.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:27 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:28:28.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:27 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:28:28.084 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:27 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:28:29.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:29 smithi162 ceph-mon[37680]: pgmap v1246: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:29.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:29 smithi162 ceph-mon[26714]: pgmap v1246: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:29.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:29 smithi181 ceph-mon[30237]: pgmap v1246: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:30.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:28:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:28:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:28:31.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:31 smithi162 ceph-mon[26714]: pgmap v1247: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:31.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:31 smithi162 ceph-mon[37680]: pgmap v1247: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:31.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:31 smithi181 ceph-mon[30237]: pgmap v1247: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:33.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:28:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:28:33.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:33 smithi181 ceph-mon[30237]: pgmap v1248: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:33.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:33 smithi162 ceph-mon[37680]: pgmap v1248: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:33.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:33 smithi162 ceph-mon[26714]: pgmap v1248: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:34.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:34 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:34.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:34 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:34.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:34 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:35.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:35 smithi162 ceph-mon[37680]: pgmap v1249: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:35.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:35 smithi162 ceph-mon[26714]: pgmap v1249: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:35.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:35 smithi181 ceph-mon[30237]: pgmap v1249: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:36.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:36.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:36.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:37.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:37 smithi162 ceph-mon[37680]: pgmap v1250: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:37.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:37 smithi162 ceph-mon[26714]: pgmap v1250: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:37.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:37 smithi181 ceph-mon[30237]: pgmap v1250: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:39.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:39 smithi162 ceph-mon[37680]: pgmap v1251: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:39.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:39 smithi162 ceph-mon[26714]: pgmap v1251: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:39.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:39 smithi181 ceph-mon[30237]: pgmap v1251: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:40.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:28:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:28:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:28:41.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:41 smithi162 ceph-mon[37680]: pgmap v1252: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:41.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:41 smithi162 ceph-mon[26714]: pgmap v1252: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:41.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:41 smithi181 ceph-mon[30237]: pgmap v1252: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:43.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:28:43 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:28:43.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:43 smithi181 ceph-mon[30237]: pgmap v1253: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:43.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:43 smithi162 ceph-mon[37680]: pgmap v1253: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:43.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:43 smithi162 ceph-mon[26714]: pgmap v1253: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:44.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:44 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:44.405 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:44 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:44.405 INFO:tasks.workunit.client.0.smithi162.stderr:++ seq 1 10 2024-04-27T01:28:44.405 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:28:44.405 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 0d36d9ee-268f-4098-865d-4f6f3764d7d6 put obj1 /etc/passwd 2024-04-27T01:28:44.430 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:28:44.431 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 0d36d9ee-268f-4098-865d-4f6f3764d7d6 put obj2 /etc/passwd 2024-04-27T01:28:44.466 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:28:44.466 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 0d36d9ee-268f-4098-865d-4f6f3764d7d6 put obj3 /etc/passwd 2024-04-27T01:28:44.501 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:28:44.501 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 0d36d9ee-268f-4098-865d-4f6f3764d7d6 put obj4 /etc/passwd 2024-04-27T01:28:44.505 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:44 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:28:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 0d36d9ee-268f-4098-865d-4f6f3764d7d6 put obj5 /etc/passwd 2024-04-27T01:28:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:28:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 0d36d9ee-268f-4098-865d-4f6f3764d7d6 put obj6 /etc/passwd 2024-04-27T01:28:44.599 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:28:44.599 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 0d36d9ee-268f-4098-865d-4f6f3764d7d6 put obj7 /etc/passwd 2024-04-27T01:28:44.631 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:28:44.632 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 0d36d9ee-268f-4098-865d-4f6f3764d7d6 put obj8 /etc/passwd 2024-04-27T01:28:44.665 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:28:44.665 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 0d36d9ee-268f-4098-865d-4f6f3764d7d6 put obj9 /etc/passwd 2024-04-27T01:28:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:+ for f in `seq 1 10` 2024-04-27T01:28:44.696 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 0d36d9ee-268f-4098-865d-4f6f3764d7d6 put obj10 /etc/passwd 2024-04-27T01:28:44.729 INFO:tasks.workunit.client.0.smithi162.stderr:+ sleep 30 2024-04-27T01:28:45.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:45 smithi162 ceph-mon[37680]: pgmap v1254: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:45.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:45 smithi162 ceph-mon[26714]: pgmap v1254: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:45.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:45 smithi181 ceph-mon[30237]: pgmap v1254: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:28:47.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:47 smithi162 ceph-mon[37680]: pgmap v1255: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:47.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:47 smithi162 ceph-mon[26714]: pgmap v1255: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:47.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:47 smithi181 ceph-mon[30237]: pgmap v1255: 188 pgs: 188 active+clean; 745 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:28:49.333 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:49 smithi162 ceph-mon[26714]: pgmap v1256: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-04-27T01:28:49.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:49 smithi162 ceph-mon[37680]: pgmap v1256: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-04-27T01:28:49.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:49 smithi181 ceph-mon[30237]: pgmap v1256: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-04-27T01:28:50.075 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:28:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:28:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:28:50.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:50 smithi162 ceph-mon[37680]: pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' is full (reached quota's max_objects: 10) 2024-04-27T01:28:50.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:50 smithi162 ceph-mon[37680]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:28:50.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:50 smithi162 ceph-mon[37680]: osdmap e779: 8 total, 8 up, 8 in 2024-04-27T01:28:50.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:50 smithi162 ceph-mon[26714]: pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' is full (reached quota's max_objects: 10) 2024-04-27T01:28:50.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:50 smithi162 ceph-mon[26714]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:28:50.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:50 smithi162 ceph-mon[26714]: osdmap e779: 8 total, 8 up, 8 in 2024-04-27T01:28:50.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:50 smithi181 ceph-mon[30237]: pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' is full (reached quota's max_objects: 10) 2024-04-27T01:28:50.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:50 smithi181 ceph-mon[30237]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-27T01:28:50.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:50 smithi181 ceph-mon[30237]: osdmap e779: 8 total, 8 up, 8 in 2024-04-27T01:28:51.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:51 smithi162 ceph-mon[37680]: pgmap v1258: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:51.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:51 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:28:51.334 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:51 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:51.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:51 smithi162 ceph-mon[26714]: pgmap v1258: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:51.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:51 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:28:51.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:51 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:51.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:51 smithi181 ceph-mon[30237]: pgmap v1258: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:51.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:51 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:28:51.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:51 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:28:53.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:28:53 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:28:53.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:53 smithi181 ceph-mon[30237]: pgmap v1259: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:53.333 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:53 smithi162 ceph-mon[37680]: pgmap v1259: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:53.334 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:53 smithi162 ceph-mon[26714]: pgmap v1259: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:54.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:54 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:54.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:54 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:54.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:54 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:28:55.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:55 smithi181 ceph-mon[30237]: pgmap v1260: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:55.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:55 smithi162 ceph-mon[37680]: pgmap v1260: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:55.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:55 smithi162 ceph-mon[26714]: pgmap v1260: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:57.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:57 smithi181 ceph-mon[30237]: pgmap v1261: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:57.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:57 smithi162 ceph-mon[37680]: pgmap v1261: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:57.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:57 smithi162 ceph-mon[26714]: pgmap v1261: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-04-27T01:28:59.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:28:59 smithi181 ceph-mon[30237]: pgmap v1262: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:28:59.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:28:59 smithi162 ceph-mon[37680]: pgmap v1262: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:28:59.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:28:59 smithi162 ceph-mon[26714]: pgmap v1262: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:29:00.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:28:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:28:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:29:01.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:01 smithi181 ceph-mon[30237]: pgmap v1263: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-04-27T01:29:01.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:01 smithi162 ceph-mon[37680]: pgmap v1263: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-04-27T01:29:01.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:01 smithi162 ceph-mon[26714]: pgmap v1263: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1016 B/s rd, 0 op/s 2024-04-27T01:29:03.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:03 smithi181 ceph-mon[30237]: pgmap v1264: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:03.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:29:03 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:29:03.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:03 smithi162 ceph-mon[37680]: pgmap v1264: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:03.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:03 smithi162 ceph-mon[26714]: pgmap v1264: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:04.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:04 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:04.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:04 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:04.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:04 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:05.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:05 smithi181 ceph-mon[30237]: pgmap v1265: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:05.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:05 smithi162 ceph-mon[37680]: pgmap v1265: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:05.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:05 smithi162 ceph-mon[26714]: pgmap v1265: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:06.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:06 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:29:06.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:06 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:29:06.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:06 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:29:07.505 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:07 smithi181 ceph-mon[30237]: pgmap v1266: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:07.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:07 smithi162 ceph-mon[37680]: pgmap v1266: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:07.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:07 smithi162 ceph-mon[26714]: pgmap v1266: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:09.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:09 smithi181 ceph-mon[30237]: pgmap v1267: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:09.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:09 smithi162 ceph-mon[37680]: pgmap v1267: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:09.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:09 smithi162 ceph-mon[26714]: pgmap v1267: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:10.083 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:09 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:29:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:29:11.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:11 smithi181 ceph-mon[30237]: pgmap v1268: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:11.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:11 smithi162 ceph-mon[37680]: pgmap v1268: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:11.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:11 smithi162 ceph-mon[26714]: pgmap v1268: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:13.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:29:13 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:29:13.256 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:13 smithi181 ceph-mon[30237]: pgmap v1269: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:13.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:13 smithi162 ceph-mon[37680]: pgmap v1269: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:13.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:13 smithi162 ceph-mon[26714]: pgmap v1269: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:14.485 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:14 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:14.485 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:14 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:14.505 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:14 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:14.730 INFO:tasks.workunit.client.0.smithi162.stderr:+ rados -p 82d3730a-02a2-4166-9bba-a35fc501ddd8 put threemore /etc/passwd 2024-04-27T01:29:14.769 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool set-quota 82d3730a-02a2-4166-9bba-a35fc501ddd8 max_bytes 0 2024-04-27T01:29:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.872+0000 7f2552dec640 1 -- 172.21.15.162:0/1510224770 >> v1:172.21.15.162:6789/0 conn(0x7f254c107690 legacy=0x7f254c107a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.872+0000 7f2552dec640 1 -- 172.21.15.162:0/1510224770 shutdown_connections 2024-04-27T01:29:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.872+0000 7f2552dec640 1 -- 172.21.15.162:0/1510224770 >> 172.21.15.162:0/1510224770 conn(0x7f254c0fd060 msgr2=0x7f254c0ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:29:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.872+0000 7f2552dec640 1 -- 172.21.15.162:0/1510224770 shutdown_connections 2024-04-27T01:29:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.872+0000 7f2552dec640 1 -- 172.21.15.162:0/1510224770 wait complete. 2024-04-27T01:29:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.872+0000 7f2552dec640 1 Processor -- start 2024-04-27T01:29:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.872+0000 7f2552dec640 1 -- start start 2024-04-27T01:29:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.872+0000 7f2552dec640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f254c1a66a0 con 0x7f254c107690 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2552dec640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f254c1a78e0 con 0x7f254c10a430 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2552dec640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f254c1a8b20 con 0x7f254c10e030 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2550b88640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f254c107690 0x7f254c106a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:52068/0 (socket says 172.21.15.162:52068) 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2550b88640 1 -- 172.21.15.162:0/3411686062 learned_addr learned my addr 172.21.15.162:0/3411686062 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447126552 0 0) 0x7f254c1a8b20 con 0x7f254c10e030 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2534003520 con 0x7f254c10e030 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2635001040 0 0) 0x7f254c1a66a0 con 0x7f254c107690 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f254c1a8b20 con 0x7f254c107690 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 132809455 0 0) 0x7f2534003520 con 0x7f254c10e030 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f254c1a66a0 con 0x7f254c10e030 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f253c0010f0 con 0x7f254c10e030 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 181598671 0 0) 0x7f254c1a8b20 con 0x7f254c107690 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2534003520 con 0x7f254c107690 2024-04-27T01:29:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 522142311 0 0) 0x7f254c1a78e0 con 0x7f254c10a430 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f254c1a8b20 con 0x7f254c10a430 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f25440048b0 con 0x7f254c107690 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4240015910 0 0) 0x7f254c1a66a0 con 0x7f254c10e030 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 >> v1:172.21.15.181:6789/0 conn(0x7f254c10a430 legacy=0x7f254c1a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 >> v1:172.21.15.162:6789/0 conn(0x7f254c107690 legacy=0x7f254c106a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254c1a9d60 con 0x7f254c10e030 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f254c1a68d0 con 0x7f254c10e030 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f253c003a80 con 0x7f254c10e030 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.873+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f253c004900 con 0x7f254c10e030 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.874+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f254c1a6da0 con 0x7f254c10e030 2024-04-27T01:29:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.874+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f254c102cf0 con 0x7f254c10e030 2024-04-27T01:29:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.875+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f253c0035f0 con 0x7f254c10e030 2024-04-27T01:29:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.875+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(779..779 src has 1..779) ==== 8142+0+0 (unknown 520171937 0 0) 0x7f253c088b30 con 0x7f254c10e030 2024-04-27T01:29:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.875+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=780}) -- 0x7f254c1a66a0 con 0x7f254c10e030 2024-04-27T01:29:14.865 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:14.880+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f253c055b20 con 0x7f254c10e030 2024-04-27T01:29:15.041 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:15.056+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"} v 0) -- 0x7f254c068880 con 0x7f254c10e030 2024-04-27T01:29:15.127 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:15.141+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 10 ==== osd_map(780..780 src has 1..780) ==== 628+0+0 (unknown 2651847096 0 0) 0x7f253c0867d0 con 0x7f254c10e030 2024-04-27T01:29:15.127 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:15.141+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=781}) -- 0x7f2534003520 con 0x7f254c10e030 2024-04-27T01:29:15.131 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:15.145+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v780) ==== 217+0+0 (unknown 3529762972 0 0) 0x7f254c068880 con 0x7f254c10e030 2024-04-27T01:29:15.238 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:15.253+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"} v 0) -- 0x7f254c105020 con 0x7f254c10e030 2024-04-27T01:29:15.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:15 smithi181 ceph-mon[30237]: pgmap v1270: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:15.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:15 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3411686062' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:15.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:15 smithi181 ceph-mon[30237]: from='client.57350 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:15.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:15 smithi162 ceph-mon[37680]: pgmap v1270: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:15.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:15 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3411686062' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:15.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:15 smithi162 ceph-mon[37680]: from='client.57350 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:15.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:15 smithi162 ceph-mon[26714]: pgmap v1270: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:15.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:15 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3411686062' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:15.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:15 smithi162 ceph-mon[26714]: from='client.57350 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:16.142 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.158+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 12 ==== osd_map(781..781 src has 1..781) ==== 628+0+0 (unknown 2427148990 0 0) 0x7f253c051460 con 0x7f254c10e030 2024-04-27T01:29:16.142 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.158+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=782}) -- 0x7f254c1a78e0 con 0x7f254c10e030 2024-04-27T01:29:16.148 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.164+0000 7f2549ffb640 1 -- 172.21.15.162:0/3411686062 <== mon.2 v1:172.21.15.162:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v781) ==== 217+0+0 (unknown 54060370 0 0) 0x7f253c0597d0 con 0x7f254c10e030 2024-04-27T01:29:16.148 INFO:tasks.workunit.client.0.smithi162.stderr:set-quota max_bytes = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 2024-04-27T01:29:16.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.167+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 >> v1:172.21.15.162:6800/624703537 conn(0x7f253406ed70 legacy=0x7f2534071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:16.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.167+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 >> v1:172.21.15.162:6790/0 conn(0x7f254c10e030 legacy=0x7f254c1a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:16.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.167+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 shutdown_connections 2024-04-27T01:29:16.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.167+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 >> 172.21.15.162:0/3411686062 conn(0x7f254c0fd060 msgr2=0x7f254c0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:29:16.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.167+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 shutdown_connections 2024-04-27T01:29:16.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.167+0000 7f2552dec640 1 -- 172.21.15.162:0/3411686062 wait complete. 2024-04-27T01:29:16.162 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool set-quota 82d3730a-02a2-4166-9bba-a35fc501ddd8 max_objects 0 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.265+0000 7f3cdbf77640 1 -- 172.21.15.162:0/2620459463 >> v1:172.21.15.162:6789/0 conn(0x7f3cd410cf40 legacy=0x7f3cd410f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.265+0000 7f3cdbf77640 1 -- 172.21.15.162:0/2620459463 shutdown_connections 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.265+0000 7f3cdbf77640 1 -- 172.21.15.162:0/2620459463 >> 172.21.15.162:0/2620459463 conn(0x7f3cd40fbfb0 msgr2=0x7f3cd40fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.265+0000 7f3cdbf77640 1 -- 172.21.15.162:0/2620459463 shutdown_connections 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.265+0000 7f3cdbf77640 1 -- 172.21.15.162:0/2620459463 wait complete. 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.266+0000 7f3cdbf77640 1 Processor -- start 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.266+0000 7f3cdbf77640 1 -- start start 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.266+0000 7f3cdbf77640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3cd41a55e0 con 0x7f3cd410cf40 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.266+0000 7f3cdbf77640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3cd41a6820 con 0x7f3cd4109340 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.266+0000 7f3cdbf77640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f3cd41a7a60 con 0x7f3cd41065a0 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.266+0000 7f3cd9d13640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f3cd41065a0 0x7f3cd4105980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:51498/0 (socket says 172.21.15.162:51498) 2024-04-27T01:29:16.251 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.266+0000 7f3cd9d13640 1 -- 172.21.15.162:0/3376537485 learned_addr learned my addr 172.21.15.162:0/3376537485 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:29:16.253 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 554877709 0 0) 0x7f3cd41a7a60 con 0x7f3cd41065a0 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc0003160 con 0x7f3cd41065a0 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3993633261 0 0) 0x7f3cd41a55e0 con 0x7f3cd410cf40 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cd41a7a60 con 0x7f3cd410cf40 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617368659 0 0) 0x7f3cd41a6820 con 0x7f3cd4109340 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cd41a55e0 con 0x7f3cd4109340 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3980326669 0 0) 0x7f3cc0003160 con 0x7f3cd41065a0 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cd41a6820 con 0x7f3cd41065a0 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f3cc80019e0 con 0x7f3cd41065a0 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2297546341 0 0) 0x7f3cd41a7a60 con 0x7f3cd410cf40 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3cc0003160 con 0x7f3cd410cf40 2024-04-27T01:29:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f3ccc001b80 con 0x7f3cd410cf40 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2259001214 0 0) 0x7f3cd41a6820 con 0x7f3cd41065a0 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 >> v1:172.21.15.181:6789/0 conn(0x7f3cd4109340 legacy=0x7f3cd41a03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 >> v1:172.21.15.162:6789/0 conn(0x7f3cd410cf40 legacy=0x7f3cd41a3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd41a8ca0 con 0x7f3cd41065a0 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd41a5810 con 0x7f3cd41065a0 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3cd41a5ce0 con 0x7f3cd41065a0 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f3cc8001cb0 con 0x7f3cd41065a0 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.267+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f3cc8005c80 con 0x7f3cd41065a0 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.268+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb8005080 con 0x7f3cd41065a0 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.268+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f3cc80038a0 con 0x7f3cd41065a0 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.269+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(781..781 src has 1..781) ==== 8142+0+0 (unknown 254895833 0 0) 0x7f3cc8089b00 con 0x7f3cd41065a0 2024-04-27T01:29:16.255 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.269+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=782}) -- 0x7f3cd41a6820 con 0x7f3cd41065a0 2024-04-27T01:29:16.258 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.273+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f3cc8056bf0 con 0x7f3cd41065a0 2024-04-27T01:29:16.436 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:16.452+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"} v 0) -- 0x7f3cb8005370 con 0x7f3cd41065a0 2024-04-27T01:29:16.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:16 smithi181 ceph-mon[30237]: from='client.57350 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:29:16.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:16 smithi181 ceph-mon[30237]: osdmap e780: 8 total, 8 up, 8 in 2024-04-27T01:29:16.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:16 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3411686062' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:16.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:16 smithi181 ceph-mon[30237]: from='client.57350 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:16.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:16 smithi162 ceph-mon[37680]: from='client.57350 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:29:16.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:16 smithi162 ceph-mon[37680]: osdmap e780: 8 total, 8 up, 8 in 2024-04-27T01:29:16.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:16 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3411686062' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:16.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:16 smithi162 ceph-mon[37680]: from='client.57350 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:16.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:16 smithi162 ceph-mon[26714]: from='client.57350 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:29:16.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:16 smithi162 ceph-mon[26714]: osdmap e780: 8 total, 8 up, 8 in 2024-04-27T01:29:16.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:16 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3411686062' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:16.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:16 smithi162 ceph-mon[26714]: from='client.57350 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]: dispatch 2024-04-27T01:29:17.150 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:17.166+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 10 ==== osd_map(782..782 src has 1..782) ==== 628+0+0 (unknown 516324702 0 0) 0x7f3cc8052ad0 con 0x7f3cd41065a0 2024-04-27T01:29:17.150 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:17.166+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=783}) -- 0x7f3cd41a55e0 con 0x7f3cd41065a0 2024-04-27T01:29:17.153 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:17.169+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v782) ==== 221+0+0 (unknown 142369548 0 0) 0x7f3cc805a8a0 con 0x7f3cd41065a0 2024-04-27T01:29:17.264 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:17.280+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"} v 0) -- 0x7f3cb8001150 con 0x7f3cd41065a0 2024-04-27T01:29:17.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:17 smithi181 ceph-mon[30237]: pgmap v1272: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:29:17.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:17 smithi181 ceph-mon[30237]: from='client.57350 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:29:17.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:17 smithi181 ceph-mon[30237]: osdmap e781: 8 total, 8 up, 8 in 2024-04-27T01:29:17.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:17 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3376537485' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:17.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:17 smithi181 ceph-mon[30237]: from='client.57356 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:17.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:17 smithi162 ceph-mon[37680]: pgmap v1272: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:29:17.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:17 smithi162 ceph-mon[37680]: from='client.57350 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:29:17.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:17 smithi162 ceph-mon[37680]: osdmap e781: 8 total, 8 up, 8 in 2024-04-27T01:29:17.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:17 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3376537485' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:17.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:17 smithi162 ceph-mon[37680]: from='client.57356 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:17.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:17 smithi162 ceph-mon[26714]: pgmap v1272: 188 pgs: 188 active+clean; 861 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:29:17.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:17 smithi162 ceph-mon[26714]: from='client.57350 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_bytes", "val": "0"}]': finished 2024-04-27T01:29:17.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:17 smithi162 ceph-mon[26714]: osdmap e781: 8 total, 8 up, 8 in 2024-04-27T01:29:17.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:17 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3376537485' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:17.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:17 smithi162 ceph-mon[26714]: from='client.57356 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:18.156 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:18.171+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 12 ==== osd_map(783..783 src has 1..783) ==== 628+0+0 (unknown 908966638 0 0) 0x7f3cc8055e50 con 0x7f3cd41065a0 2024-04-27T01:29:18.156 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:18.171+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=784}) -- 0x7f3cd41a7a60 con 0x7f3cd41065a0 2024-04-27T01:29:18.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:18.177+0000 7f3cc6ffd640 1 -- 172.21.15.162:0/3376537485 <== mon.2 v1:172.21.15.162:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 v783) ==== 221+0+0 (unknown 2284630765 0 0) 0x7f3cc805f160 con 0x7f3cd41065a0 2024-04-27T01:29:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:set-quota max_objects = 0 for pool 82d3730a-02a2-4166-9bba-a35fc501ddd8 2024-04-27T01:29:18.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:18.181+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 >> v1:172.21.15.162:6800/624703537 conn(0x7f3cc0070e10 legacy=0x7f3cc00732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:18.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:18.181+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 >> v1:172.21.15.162:6790/0 conn(0x7f3cd41065a0 legacy=0x7f3cd4105980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:18.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:18.181+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 shutdown_connections 2024-04-27T01:29:18.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:18.181+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 >> 172.21.15.162:0/3376537485 conn(0x7f3cd40fbfb0 msgr2=0x7f3cd41101a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:29:18.166 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:18.181+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 shutdown_connections 2024-04-27T01:29:18.166 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:18.181+0000 7f3cdbf77640 1 -- 172.21.15.162:0/3376537485 wait complete. 2024-04-27T01:29:18.176 INFO:tasks.workunit.client.0.smithi162.stderr:+ sleep 30 2024-04-27T01:29:18.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:18 smithi181 ceph-mon[30237]: from='client.57356 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:29:18.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:18 smithi181 ceph-mon[30237]: osdmap e782: 8 total, 8 up, 8 in 2024-04-27T01:29:18.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:18 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3376537485' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:18.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:18 smithi181 ceph-mon[30237]: from='client.57356 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:18.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:18 smithi181 ceph-mon[30237]: pgmap v1275: 188 pgs: 188 active+clean; 872 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-04-27T01:29:18.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:18 smithi162 ceph-mon[37680]: from='client.57356 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:29:18.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:18 smithi162 ceph-mon[37680]: osdmap e782: 8 total, 8 up, 8 in 2024-04-27T01:29:18.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:18 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3376537485' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:18.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:18 smithi162 ceph-mon[37680]: from='client.57356 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:18.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:18 smithi162 ceph-mon[37680]: pgmap v1275: 188 pgs: 188 active+clean; 872 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-04-27T01:29:18.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:18 smithi162 ceph-mon[26714]: from='client.57356 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:29:18.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:18 smithi162 ceph-mon[26714]: osdmap e782: 8 total, 8 up, 8 in 2024-04-27T01:29:18.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:18 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3376537485' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:18.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:18 smithi162 ceph-mon[26714]: from='client.57356 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]: dispatch 2024-04-27T01:29:18.585 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:18 smithi162 ceph-mon[26714]: pgmap v1275: 188 pgs: 188 active+clean; 872 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 1 op/s 2024-04-27T01:29:19.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:19 smithi181 ceph-mon[30237]: from='client.57356 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:29:19.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:19 smithi181 ceph-mon[30237]: osdmap e783: 8 total, 8 up, 8 in 2024-04-27T01:29:19.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:19 smithi162 ceph-mon[37680]: from='client.57356 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:29:19.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:19 smithi162 ceph-mon[37680]: osdmap e783: 8 total, 8 up, 8 in 2024-04-27T01:29:19.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:19 smithi162 ceph-mon[26714]: from='client.57356 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "field": "max_objects", "val": "0"}]': finished 2024-04-27T01:29:19.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:19 smithi162 ceph-mon[26714]: osdmap e783: 8 total, 8 up, 8 in 2024-04-27T01:29:20.086 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:19 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:29:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:29:20.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:20 smithi181 ceph-mon[30237]: pgmap v1277: 188 pgs: 188 active+clean; 872 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.8 KiB/s wr, 1 op/s 2024-04-27T01:29:20.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:20 smithi162 ceph-mon[37680]: pgmap v1277: 188 pgs: 188 active+clean; 872 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.8 KiB/s wr, 1 op/s 2024-04-27T01:29:20.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:20 smithi162 ceph-mon[26714]: pgmap v1277: 188 pgs: 188 active+clean; 872 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.8 KiB/s wr, 1 op/s 2024-04-27T01:29:21.505 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:21 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:29:21.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:21 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:29:21.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:21 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:29:22.505 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:22 smithi181 ceph-mon[30237]: pgmap v1278: 188 pgs: 188 active+clean; 872 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-04-27T01:29:22.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:22 smithi162 ceph-mon[37680]: pgmap v1278: 188 pgs: 188 active+clean; 872 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-04-27T01:29:22.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:22 smithi162 ceph-mon[26714]: pgmap v1278: 188 pgs: 188 active+clean; 872 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-04-27T01:29:23.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:23 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:23.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:29:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:29:23.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:23 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:23.584 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:23 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:24.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:24 smithi181 ceph-mon[30237]: pgmap v1279: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-04-27T01:29:24.583 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:24 smithi162 ceph-mon[26714]: pgmap v1279: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-04-27T01:29:24.584 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:24 smithi162 ceph-mon[37680]: pgmap v1279: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-04-27T01:29:26.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:26 smithi181 ceph-mon[30237]: pgmap v1280: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2024-04-27T01:29:26.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:26 smithi162 ceph-mon[26714]: pgmap v1280: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2024-04-27T01:29:26.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:26 smithi162 ceph-mon[37680]: pgmap v1280: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2024-04-27T01:29:27.755 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:27 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:29:27.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:27 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:29:27.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:27 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T01:29:28.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:28 smithi181 ceph-mon[30237]: pgmap v1281: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:29:28.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:28 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:29:28.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:28 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:29:28.756 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:28 smithi181 ceph-mon[30237]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:29:28.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:28 smithi162 ceph-mon[37680]: pgmap v1281: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:29:28.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:28 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:29:28.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:28 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:29:28.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:28 smithi162 ceph-mon[37680]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:29:28.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:28 smithi162 ceph-mon[26714]: pgmap v1281: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:29:28.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:28 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T01:29:28.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:28 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T01:29:28.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:28 smithi162 ceph-mon[26714]: from='mgr.14622 ' entity='mgr.y' 2024-04-27T01:29:30.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:29 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:29:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:29:30.755 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:30 smithi181 ceph-mon[30237]: pgmap v1282: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-04-27T01:29:30.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:30 smithi162 ceph-mon[37680]: pgmap v1282: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-04-27T01:29:30.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:30 smithi162 ceph-mon[26714]: pgmap v1282: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-04-27T01:29:32.755 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:32 smithi181 ceph-mon[30237]: pgmap v1283: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:32.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:32 smithi162 ceph-mon[37680]: pgmap v1283: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:32.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:32 smithi162 ceph-mon[26714]: pgmap v1283: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:33.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:29:33 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:29:33.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:33 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:33.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:33 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:34.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:33 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:34.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:34 smithi162 ceph-mon[26714]: pgmap v1284: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:34.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:34 smithi162 ceph-mon[37680]: pgmap v1284: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:35.005 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:34 smithi181 ceph-mon[30237]: pgmap v1284: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:36.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:36 smithi162 ceph-mon[37680]: pgmap v1285: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:36.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:36 smithi162 ceph-mon[37680]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:29:36.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:36 smithi162 ceph-mon[26714]: pgmap v1285: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:36.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:36 smithi162 ceph-mon[26714]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:29:37.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:36 smithi181 ceph-mon[30237]: pgmap v1285: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:37.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:36 smithi181 ceph-mon[30237]: from='mgr.14622 v1:172.21.15.162:0/2330565863' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-27T01:29:38.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:38 smithi162 ceph-mon[26714]: pgmap v1286: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:38.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:38 smithi162 ceph-mon[37680]: pgmap v1286: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:39.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:38 smithi181 ceph-mon[30237]: pgmap v1286: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:40.083 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:39 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:29:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:29:40.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:40 smithi162 ceph-mon[26714]: pgmap v1287: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:40.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:40 smithi162 ceph-mon[37680]: pgmap v1287: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:41.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:40 smithi181 ceph-mon[30237]: pgmap v1287: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:42.833 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:42 smithi162 ceph-mon[26714]: pgmap v1288: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:42.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:42 smithi162 ceph-mon[37680]: pgmap v1288: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:43.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:42 smithi181 ceph-mon[30237]: pgmap v1288: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:43.505 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:29:43 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:29:43.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:43 smithi162 ceph-mon[37680]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:43.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:43 smithi162 ceph-mon[26714]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:44.005 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:43 smithi181 ceph-mon[30237]: from='client.14580 v1:172.21.15.181:0/2475190927' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-27T01:29:44.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:44 smithi162 ceph-mon[37680]: pgmap v1289: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:44.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:44 smithi162 ceph-mon[26714]: pgmap v1289: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:45.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:44 smithi181 ceph-mon[30237]: pgmap v1289: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:46.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:46 smithi162 ceph-mon[37680]: pgmap v1290: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:46.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:46 smithi162 ceph-mon[26714]: pgmap v1290: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:47.005 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:46 smithi181 ceph-mon[30237]: pgmap v1290: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-27T01:29:48.177 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool delete 82d3730a-02a2-4166-9bba-a35fc501ddd8 82d3730a-02a2-4166-9bba-a35fc501ddd8 --yes-i-really-really-mean-it 2024-04-27T01:29:48.269 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.285+0000 7fd18277a640 1 -- 172.21.15.162:0/1075924033 >> v1:172.21.15.162:6789/0 conn(0x7fd17c107690 legacy=0x7fd17c107a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:48.270 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.285+0000 7fd18277a640 1 -- 172.21.15.162:0/1075924033 shutdown_connections 2024-04-27T01:29:48.270 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.285+0000 7fd18277a640 1 -- 172.21.15.162:0/1075924033 >> 172.21.15.162:0/1075924033 conn(0x7fd17c0fd060 msgr2=0x7fd17c0ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:29:48.270 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.285+0000 7fd18277a640 1 -- 172.21.15.162:0/1075924033 shutdown_connections 2024-04-27T01:29:48.270 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.285+0000 7fd18277a640 1 -- 172.21.15.162:0/1075924033 wait complete. 2024-04-27T01:29:48.270 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.285+0000 7fd18277a640 1 Processor -- start 2024-04-27T01:29:48.270 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.285+0000 7fd18277a640 1 -- start start 2024-04-27T01:29:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd18277a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd17c1a8900 con 0x7fd17c107690 2024-04-27T01:29:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd18277a640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd17c1a9b40 con 0x7fd17c10a430 2024-04-27T01:29:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd18277a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7fd17c1aad80 con 0x7fd17c10e030 2024-04-27T01:29:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd17bfff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fd17c107690 0x7fd17c106a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:59828/0 (socket says 172.21.15.162:59828) 2024-04-27T01:29:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd17bfff640 1 -- 172.21.15.162:0/3939167811 learned_addr learned my addr 172.21.15.162:0/3939167811 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:29:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2365324099 0 0) 0x7fd17c1a8900 con 0x7fd17c107690 2024-04-27T01:29:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd164003520 con 0x7fd17c107690 2024-04-27T01:29:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1157527059 0 0) 0x7fd17c1aad80 con 0x7fd17c10e030 2024-04-27T01:29:48.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd17c1a8900 con 0x7fd17c10e030 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1605077793 0 0) 0x7fd17c1a9b40 con 0x7fd17c10a430 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd17c1aad80 con 0x7fd17c10a430 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 470018369 0 0) 0x7fd17c1a8900 con 0x7fd17c10e030 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd17c1a9b40 con 0x7fd17c10e030 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd174001470 con 0x7fd17c10e030 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 664365243 0 0) 0x7fd164003520 con 0x7fd17c107690 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd17c1a8900 con 0x7fd17c107690 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd16c0043d0 con 0x7fd17c107690 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.1 v1:172.21.15.181:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4202922213 0 0) 0x7fd17c1aad80 con 0x7fd17c10a430 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.181:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd164003520 con 0x7fd17c10a430 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.286+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.1 v1:172.21.15.181:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd170001870 con 0x7fd17c10a430 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.287+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3464333366 0 0) 0x7fd17c1a9b40 con 0x7fd17c10e030 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.287+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 >> v1:172.21.15.181:6789/0 conn(0x7fd17c10a430 legacy=0x7fd17c1a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:48.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.287+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 >> v1:172.21.15.162:6789/0 conn(0x7fd17c107690 legacy=0x7fd17c106a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:48.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.287+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17c1abfc0 con 0x7fd17c10e030 2024-04-27T01:29:48.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.287+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd17c1aafb0 con 0x7fd17c10e030 2024-04-27T01:29:48.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.287+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd17c1ab480 con 0x7fd17c10e030 2024-04-27T01:29:48.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.287+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7fd174003e00 con 0x7fd17c10e030 2024-04-27T01:29:48.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.287+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7fd174004cf0 con 0x7fd17c10e030 2024-04-27T01:29:48.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.288+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd168005080 con 0x7fd17c10e030 2024-04-27T01:29:48.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.288+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7fd174003970 con 0x7fd17c10e030 2024-04-27T01:29:48.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.289+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(783..783 src has 1..783) ==== 8142+0+0 (unknown 14593009 0 0) 0x7fd174088ca0 con 0x7fd17c10e030 2024-04-27T01:29:48.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.289+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=784}) -- 0x7fd17c1a9b40 con 0x7fd17c10e030 2024-04-27T01:29:48.277 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.293+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7fd174055d90 con 0x7fd17c10e030 2024-04-27T01:29:48.454 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.469+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd168005370 con 0x7fd17c10e030 2024-04-27T01:29:48.543 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.559+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 10 ==== osd_map(784..784 src has 1..784) ==== 296+0+0 (unknown 104305979 0 0) 0x7fd174051350 con 0x7fd17c10e030 2024-04-27T01:29:48.543 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.559+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=785}) -- 0x7fd17c1a8900 con 0x7fd17c10e030 2024-04-27T01:29:48.546 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.562+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]=0 pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' removed v784) ==== 248+0+0 (unknown 1355338186 0 0) 0x7fd174059a40 con 0x7fd17c10e030 2024-04-27T01:29:48.656 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.671+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd168005da0 con 0x7fd17c10e030 2024-04-27T01:29:48.656 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.672+0000 7fd1797fa640 1 -- 172.21.15.162:0/3939167811 <== mon.2 v1:172.21.15.162:6790/0 12 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]=0 pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' does not exist v784) ==== 255+0+0 (unknown 2304851802 0 0) 0x7fd17405e300 con 0x7fd17c10e030 2024-04-27T01:29:48.656 INFO:tasks.workunit.client.0.smithi162.stderr:pool '82d3730a-02a2-4166-9bba-a35fc501ddd8' does not exist 2024-04-27T01:29:48.659 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.674+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 >> v1:172.21.15.162:6800/624703537 conn(0x7fd16406ee50 legacy=0x7fd164071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:48.659 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.674+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 >> v1:172.21.15.162:6790/0 conn(0x7fd17c10e030 legacy=0x7fd17c1a7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:48.659 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.674+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 shutdown_connections 2024-04-27T01:29:48.659 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.674+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 >> 172.21.15.162:0/3939167811 conn(0x7fd17c0fd060 msgr2=0x7fd17c0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:29:48.659 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.674+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 shutdown_connections 2024-04-27T01:29:48.659 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.674+0000 7fd18277a640 1 -- 172.21.15.162:0/3939167811 wait complete. 2024-04-27T01:29:48.667 INFO:tasks.workunit.client.0.smithi162.stderr:+ ceph osd pool delete 0d36d9ee-268f-4098-865d-4f6f3764d7d6 0d36d9ee-268f-4098-865d-4f6f3764d7d6 --yes-i-really-really-mean-it 2024-04-27T01:29:48.752 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.765+0000 7f7b78d13640 1 -- 172.21.15.162:0/3385609246 >> v1:172.21.15.162:6790/0 conn(0x7f7b7410cf40 legacy=0x7f7b7410f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:48.752 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.765+0000 7f7b78d13640 1 -- 172.21.15.162:0/3385609246 shutdown_connections 2024-04-27T01:29:48.752 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.765+0000 7f7b78d13640 1 -- 172.21.15.162:0/3385609246 >> 172.21.15.162:0/3385609246 conn(0x7f7b740fbfb0 msgr2=0x7f7b740fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:29:48.752 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.765+0000 7f7b78d13640 1 -- 172.21.15.162:0/3385609246 shutdown_connections 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.765+0000 7f7b78d13640 1 -- 172.21.15.162:0/3385609246 wait complete. 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.766+0000 7f7b78d13640 1 Processor -- start 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.766+0000 7f7b78d13640 1 -- start start 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.766+0000 7f7b78d13640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7b741a5550 con 0x7f7b741065a0 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.766+0000 7f7b78d13640 1 -- --> v1:172.21.15.181:6789/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7b741a6790 con 0x7f7b7410cf40 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.766+0000 7f7b78d13640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 34 bytes epoch 0) -- 0x7f7b741a79d0 con 0x7f7b74109340 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.766+0000 7f7b7259c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f7b741065a0 0x7f7b74105900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:59848/0 (socket says 172.21.15.162:59848) 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.766+0000 7f7b7259c640 1 -- 172.21.15.162:0/1897296540 learned_addr learned my addr 172.21.15.162:0/1897296540 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.766+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3194420026 0 0) 0x7f7b741a79d0 con 0x7f7b74109340 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.766+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b5c003520 con 0x7f7b74109340 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3477957353 0 0) 0x7f7b741a5550 con 0x7f7b741065a0 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b741a79d0 con 0x7f7b741065a0 2024-04-27T01:29:48.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 387802975 0 0) 0x7f7b5c003520 con 0x7f7b74109340 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b741a5550 con 0x7f7b74109340 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7b68001870 con 0x7f7b74109340 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2603375842 0 0) 0x7f7b741a79d0 con 0x7f7b741065a0 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7b5c003520 con 0x7f7b741065a0 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7b64001230 con 0x7f7b741065a0 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.1 v1:172.21.15.181:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 587728172 0 0) 0x7f7b741a6790 con 0x7f7b7410cf40 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.181:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b741a79d0 con 0x7f7b7410cf40 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3481385336 0 0) 0x7f7b741a5550 con 0x7f7b74109340 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 >> v1:172.21.15.181:6789/0 conn(0x7f7b7410cf40 legacy=0x7f7b741a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 >> v1:172.21.15.162:6789/0 conn(0x7f7b741065a0 legacy=0x7f7b74105900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b741a8c10 con 0x7f7b74109340 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7b741a5780 con 0x7f7b74109340 2024-04-27T01:29:48.754 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7b741a5c50 con 0x7f7b74109340 2024-04-27T01:29:48.755 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(25 keys) ==== 1068+0+0 (unknown 250026635 0 0) 0x7f7b68003b20 con 0x7f7b74109340 2024-04-27T01:29:48.755 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.767+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 2576655195 0 0) 0x7f7b68004b00 con 0x7f7b74109340 2024-04-27T01:29:48.755 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.768+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b54005080 con 0x7f7b74109340 2024-04-27T01:29:48.755 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.768+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 21) ==== 92140+0+0 (unknown 4248326459 0 0) 0x7f7b68003690 con 0x7f7b74109340 2024-04-27T01:29:48.755 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.769+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(784..784 src has 1..784) ==== 7754+0+0 (unknown 535895647 0 0) 0x7f7b68088a00 con 0x7f7b74109340 2024-04-27T01:29:48.755 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.769+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=785}) -- 0x7f7b741a5550 con 0x7f7b74109340 2024-04-27T01:29:48.757 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.773+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (unknown 1092875540 0 3304102404) 0x7f7b68055b90 con 0x7f7b74109340 2024-04-27T01:29:48.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:48 smithi162 ceph-mon[37680]: pgmap v1291: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:48.833 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:48 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3939167811' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:48.834 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:48 smithi162 ceph-mon[37680]: from='client.57362 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:48.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:48 smithi162 ceph-mon[26714]: pgmap v1291: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:48.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:48 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3939167811' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:48.834 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:48 smithi162 ceph-mon[26714]: from='client.57362 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:48.934 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:48.950+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7b54005370 con 0x7f7b74109340 2024-04-27T01:29:49.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:48 smithi181 ceph-mon[30237]: pgmap v1291: 188 pgs: 188 active+clean; 872 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-27T01:29:49.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:48 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3939167811' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:49.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:48 smithi181 ceph-mon[30237]: from='client.57362 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:49.550 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.566+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 10 ==== osd_map(785..785 src has 1..785) ==== 296+0+0 (unknown 2268356862 0 0) 0x7f7b68086840 con 0x7f7b74109340 2024-04-27T01:29:49.551 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.566+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=786}) -- 0x7f7b5c003520 con 0x7f7b74109340 2024-04-27T01:29:49.557 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.573+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]=0 pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' removed v785) ==== 248+0+0 (unknown 4029564623 0 0) 0x7f7b68059840 con 0x7f7b74109340 2024-04-27T01:29:49.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.683+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7b54005da0 con 0x7f7b74109340 2024-04-27T01:29:49.668 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.684+0000 7f7b637fe640 1 -- 172.21.15.162:0/1897296540 <== mon.2 v1:172.21.15.162:6790/0 12 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]=0 pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' does not exist v785) ==== 255+0+0 (unknown 3810109654 0 0) 0x7f7b6805e100 con 0x7f7b74109340 2024-04-27T01:29:49.668 INFO:tasks.workunit.client.0.smithi162.stderr:pool '0d36d9ee-268f-4098-865d-4f6f3764d7d6' does not exist 2024-04-27T01:29:49.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.686+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 >> v1:172.21.15.162:6800/624703537 conn(0x7f7b5c06ee00 legacy=0x7f7b5c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:49.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.686+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 >> v1:172.21.15.162:6790/0 conn(0x7f7b74109340 legacy=0x7f7b741a03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T01:29:49.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.686+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 shutdown_connections 2024-04-27T01:29:49.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.686+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 >> 172.21.15.162:0/1897296540 conn(0x7f7b740fbfb0 msgr2=0x7f7b740fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T01:29:49.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.686+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 shutdown_connections 2024-04-27T01:29:49.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-04-27T01:29:49.686+0000 7f7b78d13640 1 -- 172.21.15.162:0/1897296540 wait complete. 2024-04-27T01:29:49.680 INFO:tasks.workunit.client.0.smithi162.stdout:OK 2024-04-27T01:29:49.681 INFO:tasks.workunit.client.0.smithi162.stderr:+ echo OK 2024-04-27T01:29:49.681 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-27T01:29:49.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-27T01:29:49.745 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-04-27T01:29:49.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-27T01:29:49.805 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: ::ffff:172.21.15.181 - - [27/Apr/2024:01:29:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-27T01:29:49.805 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[37680]: from='client.57362 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T01:29:49.805 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[37680]: osdmap e784: 8 total, 8 up, 8 in 2024-04-27T01:29:49.805 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/3939167811' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:49.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[37680]: from='client.57362 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:49.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1897296540' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:49.806 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[37680]: from='client.57368 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:49.806 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[26714]: from='client.57362 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T01:29:49.806 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[26714]: osdmap e784: 8 total, 8 up, 8 in 2024-04-27T01:29:49.806 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/3939167811' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:49.806 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[26714]: from='client.57362 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:49.806 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1897296540' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:49.806 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:49 smithi162 ceph-mon[26714]: from='client.57368 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:50.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:49 smithi181 ceph-mon[30237]: from='client.57362 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T01:29:50.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:49 smithi181 ceph-mon[30237]: osdmap e784: 8 total, 8 up, 8 in 2024-04-27T01:29:50.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/3939167811' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:50.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:49 smithi181 ceph-mon[30237]: from='client.57362 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "pool2": "82d3730a-02a2-4166-9bba-a35fc501ddd8", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:50.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:49 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1897296540' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:50.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:49 smithi181 ceph-mon[30237]: from='client.57368 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:50.070 DEBUG:teuthology.parallel:result is None 2024-04-27T01:29:50.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-27T01:29:50.134 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-27T01:29:50.134 DEBUG:teuthology.orchestra.run.smithi162:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-27T01:29:50.196 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-27T01:29:50.197 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-27T01:29:50.208 INFO:tasks.cephadm:Teardown begin 2024-04-27T01:29:50.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:29:50.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:29:50.293 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-27T01:29:50.293 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T01:29:50.315 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T01:29:50.350 INFO:tasks.cephadm:Stopping all daemons... 2024-04-27T01:29:50.350 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-27T01:29:50.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.a 2024-04-27T01:29:50.663 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[37680]: pgmap v1293: 176 pgs: 176 active+clean; 699 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:29:50.663 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[37680]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:29:50.663 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[37680]: from='client.57368 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T01:29:50.663 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[37680]: osdmap e785: 8 total, 8 up, 8 in 2024-04-27T01:29:50.663 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[37680]: from='client.? v1:172.21.15.162:0/1897296540' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:50.663 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[37680]: from='client.57368 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:50.664 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:50 smithi162 systemd[1]: Stopping Ceph mon.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:29:50.664 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[26714]: pgmap v1293: 176 pgs: 176 active+clean; 699 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:29:50.664 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[26714]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:29:50.664 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[26714]: from='client.57368 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T01:29:50.664 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[26714]: osdmap e785: 8 total, 8 up, 8 in 2024-04-27T01:29:50.664 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[26714]: from='client.? v1:172.21.15.162:0/1897296540' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:50.664 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-mon[26714]: from='client.57368 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:50.939 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a[26710]: 2024-04-27T01:29:50.678+0000 7fdcc1e1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:29:50.939 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:50 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a[26710]: 2024-04-27T01:29:50.678+0000 7fdcc1e1e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-27T01:29:50.939 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:50 smithi162 podman[146280]: 2024-04-27 01:29:50.719952787 +0000 UTC m=+0.165125022 container died cc5d5433d217df96e4e2fd83d028ebf1a04bcd02ee47a0f49f909f20a02fdbb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb) 2024-04-27T01:29:51.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:50 smithi181 ceph-mon[30237]: pgmap v1293: 176 pgs: 176 active+clean; 699 KiB data, 850 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-27T01:29:51.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:50 smithi181 ceph-mon[30237]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-27T01:29:51.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:50 smithi181 ceph-mon[30237]: from='client.57368 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T01:29:51.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:50 smithi181 ceph-mon[30237]: osdmap e785: 8 total, 8 up, 8 in 2024-04-27T01:29:51.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:50 smithi181 ceph-mon[30237]: from='client.? v1:172.21.15.162:0/1897296540' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:51.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:50 smithi181 ceph-mon[30237]: from='client.57368 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "pool2": "0d36d9ee-268f-4098-865d-4f6f3764d7d6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T01:29:51.322 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:51 smithi162 podman[146280]: 2024-04-27 01:29:51.153998682 +0000 UTC m=+0.599170924 container cleanup cc5d5433d217df96e4e2fd83d028ebf1a04bcd02ee47a0f49f909f20a02fdbb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, ceph=True) 2024-04-27T01:29:51.323 INFO:journalctl@ceph.mon.a.smithi162.stdout:Apr 27 01:29:51 smithi162 bash[146280]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-a 2024-04-27T01:29:51.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.a.service' 2024-04-27T01:29:51.481 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:29:51.481 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-27T01:29:51.481 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-27T01:29:51.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.c 2024-04-27T01:29:51.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:51 smithi162 systemd[1]: Stopping Ceph mon.c for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:29:52.083 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-c[37676]: 2024-04-27T01:29:51.825+0000 7f238d148700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:29:52.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-c[37676]: 2024-04-27T01:29:51.825+0000 7f238d148700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-04-27T01:29:52.084 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:52 smithi162 podman[146403]: 2024-04-27 01:29:52.040337362 +0000 UTC m=+0.319810425 container died d8b6557e66817d0261f49cb8f1953d000782e700024885e6d8d9e8a6289dbbb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:29:52.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:29:51] ENGINE Bus STOPPING 2024-04-27T01:29:52.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:29:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-27T01:29:52.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:29:51] ENGINE Bus STOPPED 2024-04-27T01:29:52.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:29:51] ENGINE Bus STARTING 2024-04-27T01:29:52.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:29:51] ENGINE Serving on http://:::9283 2024-04-27T01:29:52.084 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:51 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: [27/Apr/2024:01:29:51] ENGINE Bus STARTED 2024-04-27T01:29:52.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:52 smithi162 podman[146403]: 2024-04-27 01:29:52.425043662 +0000 UTC m=+0.704516725 container cleanup d8b6557e66817d0261f49cb8f1953d000782e700024885e6d8d9e8a6289dbbb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-27T01:29:52.583 INFO:journalctl@ceph.mon.c.smithi162.stdout:Apr 27 01:29:52 smithi162 bash[146403]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-c 2024-04-27T01:29:52.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.c.service' 2024-04-27T01:29:52.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:29:52.858 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-27T01:29:52.858 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-27T01:29:52.858 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.b 2024-04-27T01:29:53.238 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:52 smithi181 systemd[1]: Stopping Ceph mon.b for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:29:53.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:29:53 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug there is no tcmu-runner data available 2024-04-27T01:29:53.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:53 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-b[30233]: 2024-04-27T01:29:53.253+0000 7fcfe870a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:29:53.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:53 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-b[30233]: 2024-04-27T01:29:53.253+0000 7fcfe870a700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-04-27T01:29:53.506 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:53 smithi181 podman[66509]: 2024-04-27 01:29:53.457642777 +0000 UTC m=+0.384014151 container died f4dafb30ceb6b30ebdfdf89f047e06a20400f49b8ed47abc38df32fbf3024d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2) 2024-04-27T01:29:54.005 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:53 smithi181 podman[66509]: 2024-04-27 01:29:53.853217507 +0000 UTC m=+0.779588920 container cleanup f4dafb30ceb6b30ebdfdf89f047e06a20400f49b8ed47abc38df32fbf3024d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-b, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2) 2024-04-27T01:29:54.006 INFO:journalctl@ceph.mon.b.smithi181.stdout:Apr 27 01:29:53 smithi181 bash[66509]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mon-b 2024-04-27T01:29:54.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mon.b.service' 2024-04-27T01:29:54.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:29:54.308 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-27T01:29:54.308 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-27T01:29:54.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mgr.y 2024-04-27T01:29:54.583 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:54 smithi162 systemd[1]: Stopping Ceph mgr.y for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:29:54.872 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:54 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:29:54.614+0000 7f902598f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:29:54.872 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:54 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y[26943]: 2024-04-27T01:29:54.614+0000 7f902598f700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-27T01:29:54.872 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:54 smithi162 podman[146535]: 2024-04-27 01:29:54.64679319 +0000 UTC m=+0.134198524 container died 17466af6dc5b9789b2a0234364343d07265050cd912a00354336d7df21295d95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-27T01:29:55.299 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:54 smithi162 podman[146535]: 2024-04-27 01:29:54.962428599 +0000 UTC m=+0.449833939 container cleanup 17466af6dc5b9789b2a0234364343d07265050cd912a00354336d7df21295d95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-b22e2eb, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:29:55.300 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:54 smithi162 bash[146535]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y 2024-04-27T01:29:55.300 INFO:journalctl@ceph.mgr.y.smithi162.stdout:Apr 27 01:29:55 smithi162 podman[146549]: 2024-04-27 01:29:55.105337383 +0000 UTC m=+0.457499961 container remove 17466af6dc5b9789b2a0234364343d07265050cd912a00354336d7df21295d95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b22e2ebdeb24376882b7bda2a7329c8cccc2276a, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-27T01:29:55.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-27T01:29:55.344 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:29:55.344 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-27T01:29:55.344 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-27T01:29:55.345 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mgr.x 2024-04-27T01:29:55.662 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:29:55 smithi181 systemd[1]: Stopping Ceph mgr.x for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:29:55.977 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:29:55 smithi181 podman[66632]: 2024-04-27 01:29:55.678846884 +0000 UTC m=+0.122211384 container died ef824765aa95b8b8c2760e90d29a0109f6984f73ad31ccb111904b02ec440f90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:29:56.256 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:29:56 smithi181 podman[66632]: 2024-04-27 01:29:56.08308477 +0000 UTC m=+0.526449271 container cleanup ef824765aa95b8b8c2760e90d29a0109f6984f73ad31ccb111904b02ec440f90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:29:56.256 INFO:journalctl@ceph.mgr.x.smithi181.stdout:Apr 27 01:29:56 smithi181 bash[66632]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-mgr-x 2024-04-27T01:29:56.430 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-27T01:29:56.465 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:29:56.465 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-27T01:29:56.465 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-27T01:29:56.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.0 2024-04-27T01:29:56.758 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:29:56 smithi162 systemd[1]: Stopping Ceph osd.0 for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:29:57.083 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:29:56 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0[41669]: 2024-04-27T01:29:56.772+0000 7fda6c5b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:29:57.084 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:29:56 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0[41669]: 2024-04-27T01:29:56.773+0000 7fda6c5b8700 -1 osd.0 785 *** Got signal Terminated *** 2024-04-27T01:29:57.084 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:29:56 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0[41669]: 2024-04-27T01:29:56.773+0000 7fda6c5b8700 -1 osd.0 785 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:30:02.401 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:30:02 smithi162 podman[146657]: 2024-04-27 01:30:02.143022282 +0000 UTC m=+5.472169171 container died 834c2fed8fe60317c98aa600edbf3888a6dbbb5f3767e9a36b3b30ef890b75b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-27T01:30:02.734 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:30:02 smithi162 podman[146657]: 2024-04-27 01:30:02.490959146 +0000 UTC m=+5.820106036 container cleanup 834c2fed8fe60317c98aa600edbf3888a6dbbb5f3767e9a36b3b30ef890b75b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-27T01:30:02.734 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:30:02 smithi162 bash[146657]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0 2024-04-27T01:30:03.083 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:30:02 smithi162 podman[146676]: 2024-04-27 01:30:02.750405927 +0000 UTC m=+0.609711330 container remove 834c2fed8fe60317c98aa600edbf3888a6dbbb5f3767e9a36b3b30ef890b75b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-27T01:30:03.387 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:30:03 smithi162 podman[146766]: 2024-04-27 01:30:03.117883177 +0000 UTC m=+0.122260764 container create b440dbe79be023c42e07cc38caf5764b1b1058bc63fdeae194965e33f9262e9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-27T01:30:03.387 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:30:03 smithi162 podman[146766]: 2024-04-27 01:30:03.024234057 +0000 UTC m=+0.028611601 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:30:03.388 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:30:03 smithi162 podman[146766]: 2024-04-27 01:30:03.311979425 +0000 UTC m=+0.316356972 container init b440dbe79be023c42e07cc38caf5764b1b1058bc63fdeae194965e33f9262e9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-27T01:30:03.388 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:30:03 smithi162 podman[146766]: 2024-04-27 01:30:03.315356789 +0000 UTC m=+0.319734336 container start b440dbe79be023c42e07cc38caf5764b1b1058bc63fdeae194965e33f9262e9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T01:30:03.388 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:30:03 smithi162 podman[146766]: 2024-04-27 01:30:03.403465046 +0000 UTC m=+0.407842607 container attach b440dbe79be023c42e07cc38caf5764b1b1058bc63fdeae194965e33f9262e9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-27T01:30:03.834 INFO:journalctl@ceph.osd.0.smithi162.stdout:Apr 27 01:30:03 smithi162 podman[146766]: 2024-04-27 01:30:03.617577356 +0000 UTC m=+0.621954894 container died b440dbe79be023c42e07cc38caf5764b1b1058bc63fdeae194965e33f9262e9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-27T01:30:03.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:03 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:03.578Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:30:03.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:03 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:03.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-04-27T01:30:04.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.0.service' 2024-04-27T01:30:04.207 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:30:04.207 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-27T01:30:04.207 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-27T01:30:04.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.1 2024-04-27T01:30:04.449 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:04 smithi162 systemd[1]: Stopping Ceph osd.1 for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:30:04.833 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:04 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1[47531]: 2024-04-27T01:30:04.568+0000 7fb9d5cc0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:30:04.834 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:04 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1[47531]: 2024-04-27T01:30:04.568+0000 7fb9d5cc0700 -1 osd.1 785 *** Got signal Terminated *** 2024-04-27T01:30:04.834 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:04 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1[47531]: 2024-04-27T01:30:04.568+0000 7fb9d5cc0700 -1 osd.1 785 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:30:10.262 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:10 smithi162 podman[146872]: 2024-04-27 01:30:10.015811895 +0000 UTC m=+5.549540855 container died af7d2c84272efecb4c2a2f59c8da3fc03fe6fdaae14bff09c143454279b0efe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:30:10.544 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:10 smithi162 podman[146872]: 2024-04-27 01:30:10.359408 +0000 UTC m=+5.893136960 container cleanup af7d2c84272efecb4c2a2f59c8da3fc03fe6fdaae14bff09c143454279b0efe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-27T01:30:10.545 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:10 smithi162 bash[146872]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1 2024-04-27T01:30:10.833 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:10 smithi162 podman[146903]: 2024-04-27 01:30:10.560715262 +0000 UTC m=+0.543805264 container remove af7d2c84272efecb4c2a2f59c8da3fc03fe6fdaae14bff09c143454279b0efe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-27T01:30:11.247 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:10 smithi162 podman[146992]: 2024-04-27 01:30:10.879564014 +0000 UTC m=+0.018170312 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:30:11.247 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:10 smithi162 podman[146992]: 2024-04-27 01:30:10.977911722 +0000 UTC m=+0.116517992 container create 1cc64b2913ddc1450e16c0ef9fa9d2e0336dc1a2854da7a042fb22258dfee268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:30:11.247 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:11 smithi162 podman[146992]: 2024-04-27 01:30:11.197176816 +0000 UTC m=+0.335783091 container init 1cc64b2913ddc1450e16c0ef9fa9d2e0336dc1a2854da7a042fb22258dfee268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-27T01:30:11.247 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:11 smithi162 podman[146992]: 2024-04-27 01:30:11.200664135 +0000 UTC m=+0.339270405 container start 1cc64b2913ddc1450e16c0ef9fa9d2e0336dc1a2854da7a042fb22258dfee268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-27T01:30:11.247 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:11 smithi162 podman[146992]: 2024-04-27 01:30:11.263367548 +0000 UTC m=+0.401973828 container attach 1cc64b2913ddc1450e16c0ef9fa9d2e0336dc1a2854da7a042fb22258dfee268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:30:11.583 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:11 smithi162 podman[146992]: 2024-04-27 01:30:11.490520491 +0000 UTC m=+0.629126768 container died 1cc64b2913ddc1450e16c0ef9fa9d2e0336dc1a2854da7a042fb22258dfee268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-27T01:30:12.043 INFO:journalctl@ceph.osd.1.smithi162.stdout:Apr 27 01:30:12 smithi162 podman[147018]: 2024-04-27 01:30:12.041522766 +0000 UTC m=+0.542780404 container remove 1cc64b2913ddc1450e16c0ef9fa9d2e0336dc1a2854da7a042fb22258dfee268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-04-27T01:30:12.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.1.service' 2024-04-27T01:30:12.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:30:12.089 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-27T01:30:12.090 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-27T01:30:12.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.2 2024-04-27T01:30:12.333 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:12 smithi162 systemd[1]: Stopping Ceph osd.2 for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:30:12.833 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:12 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2[53457]: 2024-04-27T01:30:12.436+0000 7f6878c7c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:30:12.834 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:12 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2[53457]: 2024-04-27T01:30:12.436+0000 7f6878c7c700 -1 osd.2 785 *** Got signal Terminated *** 2024-04-27T01:30:12.834 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:12 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2[53457]: 2024-04-27T01:30:12.436+0000 7f6878c7c700 -1 osd.2 785 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:30:13.833 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:13 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:13.578Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:30:13.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:13 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:13.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:30:13.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:13 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:13.579Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-04-27T01:30:18.296 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:17 smithi162 podman[147098]: 2024-04-27 01:30:17.983087677 +0000 UTC m=+5.657592100 container died 51bc2e2d28439ca0f2442467dae7ed9d9c4912c839d58b359eac7029136d5765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-04-27T01:30:18.583 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:18 smithi162 podman[147098]: 2024-04-27 01:30:18.385416741 +0000 UTC m=+6.059921161 container cleanup 51bc2e2d28439ca0f2442467dae7ed9d9c4912c839d58b359eac7029136d5765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T01:30:18.583 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:18 smithi162 bash[147098]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2 2024-04-27T01:30:18.937 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:18 smithi162 podman[147127]: 2024-04-27 01:30:18.636539834 +0000 UTC m=+0.648368312 container remove 51bc2e2d28439ca0f2442467dae7ed9d9c4912c839d58b359eac7029136d5765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:30:19.214 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:18 smithi162 podman[147218]: 2024-04-27 01:30:18.953948572 +0000 UTC m=+0.111932509 container create db2747eed6ca8f19f7c7ac17a7ec7edc7dd2a41f4d159b3308bef7dcbd648d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:30:19.215 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:18 smithi162 podman[147218]: 2024-04-27 01:30:18.860403349 +0000 UTC m=+0.018387256 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:30:19.215 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:19 smithi162 podman[147218]: 2024-04-27 01:30:19.147963608 +0000 UTC m=+0.305947512 container init db2747eed6ca8f19f7c7ac17a7ec7edc7dd2a41f4d159b3308bef7dcbd648d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-27T01:30:19.215 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:19 smithi162 podman[147218]: 2024-04-27 01:30:19.151506787 +0000 UTC m=+0.309490682 container start db2747eed6ca8f19f7c7ac17a7ec7edc7dd2a41f4d159b3308bef7dcbd648d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:30:19.215 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:19 smithi162 podman[147218]: 2024-04-27 01:30:19.230920501 +0000 UTC m=+0.388904402 container attach db2747eed6ca8f19f7c7ac17a7ec7edc7dd2a41f4d159b3308bef7dcbd648d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:30:19.584 INFO:journalctl@ceph.osd.2.smithi162.stdout:Apr 27 01:30:19 smithi162 podman[147218]: 2024-04-27 01:30:19.440847343 +0000 UTC m=+0.598831247 container died db2747eed6ca8f19f7c7ac17a7ec7edc7dd2a41f4d159b3308bef7dcbd648d33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-27T01:30:20.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.2.service' 2024-04-27T01:30:20.039 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:30:20.040 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-27T01:30:20.040 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-27T01:30:20.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.3 2024-04-27T01:30:20.334 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:20 smithi162 systemd[1]: Stopping Ceph osd.3 for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:30:20.833 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:20 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3[59259]: 2024-04-27T01:30:20.387+0000 7f0d1d691700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:30:20.834 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:20 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3[59259]: 2024-04-27T01:30:20.387+0000 7f0d1d691700 -1 osd.3 785 *** Got signal Terminated *** 2024-04-27T01:30:20.834 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:20 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3[59259]: 2024-04-27T01:30:20.387+0000 7f0d1d691700 -1 osd.3 785 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:30:23.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:22 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:22.812+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:23.833 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:23 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:23.579Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:30:23.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:23 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:23.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:30:23.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:23 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:23.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-04-27T01:30:24.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:23.799+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:24.006 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:23 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:23.569+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:25.005 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:24.592+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:25.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:24.772+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:25.006 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:24 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:24.713+0000 7fdab0523700 -1 osd.4 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.544483+0000 front 2024-04-27T01:30:00.544448+0000 (oldest deadline 2024-04-27T01:30:24.644565+0000) 2024-04-27T01:30:25.333 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:24 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3[59259]: 2024-04-27T01:30:24.963+0000 7f0d15c7d700 -1 osd.3 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:01.653353+0000 front 2024-04-27T01:30:01.653371+0000 (oldest deadline 2024-04-27T01:30:23.953515+0000) 2024-04-27T01:30:26.005 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:25 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:25.735+0000 7fdab0523700 -1 osd.4 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.544483+0000 front 2024-04-27T01:30:00.544448+0000 (oldest deadline 2024-04-27T01:30:24.644565+0000) 2024-04-27T01:30:26.006 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:25 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:25.571+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:26.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:25 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:25.790+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:26.204 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:25 smithi162 podman[147327]: 2024-04-27 01:30:25.956245149 +0000 UTC m=+5.680964054 container died 878aeea716aed4ecee146ab966e8edb2ee0a15bc8aba71acfa130b81e41d1dc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb) 2024-04-27T01:30:26.583 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:26 smithi162 podman[147327]: 2024-04-27 01:30:26.319396728 +0000 UTC m=+6.044115642 container cleanup 878aeea716aed4ecee146ab966e8edb2ee0a15bc8aba71acfa130b81e41d1dc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:30:26.583 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:26 smithi162 bash[147327]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3 2024-04-27T01:30:26.816 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:26.783+0000 7fdab0523700 -1 osd.4 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.544483+0000 front 2024-04-27T01:30:00.544448+0000 (oldest deadline 2024-04-27T01:30:24.644565+0000) 2024-04-27T01:30:26.816 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:26.622+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:26.816 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:26.567+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:26.918 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:26 smithi162 podman[147345]: 2024-04-27 01:30:26.603701011 +0000 UTC m=+0.648006275 container remove 878aeea716aed4ecee146ab966e8edb2ee0a15bc8aba71acfa130b81e41d1dc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424) 2024-04-27T01:30:27.215 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:26 smithi162 podman[147431]: 2024-04-27 01:30:26.837492999 +0000 UTC m=+0.017887059 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:30:27.216 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:26 smithi162 podman[147431]: 2024-04-27 01:30:26.95462038 +0000 UTC m=+0.135014489 container create 4b0f35dddfc51f0c156ea40d1efa2fecc0ba0644816840bb240a120d145c0445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-27T01:30:27.216 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:27 smithi162 podman[147431]: 2024-04-27 01:30:27.148608619 +0000 UTC m=+0.329002656 container init 4b0f35dddfc51f0c156ea40d1efa2fecc0ba0644816840bb240a120d145c0445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:30:27.216 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:27 smithi162 podman[147431]: 2024-04-27 01:30:27.151613489 +0000 UTC m=+0.332007531 container start 4b0f35dddfc51f0c156ea40d1efa2fecc0ba0644816840bb240a120d145c0445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-27T01:30:27.216 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:27 smithi162 podman[147431]: 2024-04-27 01:30:27.231829887 +0000 UTC m=+0.412223925 container attach 4b0f35dddfc51f0c156ea40d1efa2fecc0ba0644816840bb240a120d145c0445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424) 2024-04-27T01:30:27.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:26 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:26.831+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:27.583 INFO:journalctl@ceph.osd.3.smithi162.stdout:Apr 27 01:30:27 smithi162 podman[147431]: 2024-04-27 01:30:27.444569755 +0000 UTC m=+0.624963804 container died 4b0f35dddfc51f0c156ea40d1efa2fecc0ba0644816840bb240a120d145c0445 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-04-27T01:30:27.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.3.service' 2024-04-27T01:30:28.005 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:27.621+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:28.006 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:27.789+0000 7fdab0523700 -1 osd.4 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.544483+0000 front 2024-04-27T01:30:00.544448+0000 (oldest deadline 2024-04-27T01:30:24.644565+0000) 2024-04-27T01:30:28.006 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:27.610+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:28.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:27 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:27.834+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:28.029 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:30:28.029 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-27T01:30:28.030 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-27T01:30:28.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.4 2024-04-27T01:30:28.369 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:28 smithi181 systemd[1]: Stopping Ceph osd.4 for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:30:28.625 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:28.620+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:28.625 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:28.384+0000 7fdab3f2f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:30:28.625 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:28.384+0000 7fdab3f2f700 -1 osd.4 785 *** Got signal Terminated *** 2024-04-27T01:30:28.625 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:28.384+0000 7fdab3f2f700 -1 osd.4 785 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:30:29.005 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:28.746+0000 7fdab0523700 -1 osd.4 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.544483+0000 front 2024-04-27T01:30:00.544448+0000 (oldest deadline 2024-04-27T01:30:24.644565+0000) 2024-04-27T01:30:29.006 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:28.639+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:29.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:28 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:28.872+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:29.873 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:29.786+0000 7fdab0523700 -1 osd.4 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.544483+0000 front 2024-04-27T01:30:00.544448+0000 (oldest deadline 2024-04-27T01:30:24.644565+0000) 2024-04-27T01:30:29.873 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:29.595+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:29.873 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:29.584+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:30.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:29.888+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:30.895 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:30.739+0000 7fdab0523700 -1 osd.4 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.544483+0000 front 2024-04-27T01:30:00.544448+0000 (oldest deadline 2024-04-27T01:30:24.644565+0000) 2024-04-27T01:30:30.895 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:30.565+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:30.896 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:30.622+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:30.896 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:30.622+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:31.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:30.910+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:31.918 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:31 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:31.702+0000 7fdab0523700 -1 osd.4 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.544483+0000 front 2024-04-27T01:30:00.544448+0000 (oldest deadline 2024-04-27T01:30:24.644565+0000) 2024-04-27T01:30:31.918 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:31 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:31.534+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:31.919 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:31 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:31.622+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:31.919 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:31 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:31.622+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:32.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:31 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:31.934+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:32.932 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4[35166]: 2024-04-27T01:30:32.733+0000 7fdab0523700 -1 osd.4 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.544483+0000 front 2024-04-27T01:30:00.544448+0000 (oldest deadline 2024-04-27T01:30:24.644565+0000) 2024-04-27T01:30:32.933 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:32.575+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:32.933 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:32.639+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:32.933 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:32.639+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:33.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:32 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:32.947+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:33.833 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:33 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:33.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-04-27T01:30:33.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:33 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:33.580Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:30:33.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:33 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:33.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-04-27T01:30:33.890 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:33 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:33.597+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:33.890 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:33 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:33.592+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:33.891 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:33 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:33.592+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:34.204 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:33 smithi181 podman[66777]: 2024-04-27 01:30:33.906609024 +0000 UTC m=+5.660527554 container died 59ffc8a6d8e462870077bfe14752a4001995424a444968bd46867ccdd4fd63da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-27T01:30:34.204 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:33 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:33.925+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:34.495 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:34 smithi181 podman[66777]: 2024-04-27 01:30:34.318413346 +0000 UTC m=+6.072331877 container cleanup 59ffc8a6d8e462870077bfe14752a4001995424a444968bd46867ccdd4fd63da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-04-27T01:30:34.495 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:34 smithi181 bash[66777]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4 2024-04-27T01:30:34.755 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:34 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:34.614+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:34.756 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:34 smithi181 podman[66795]: 2024-04-27 01:30:34.51114687 +0000 UTC m=+0.602923247 container remove 59ffc8a6d8e462870077bfe14752a4001995424a444968bd46867ccdd4fd63da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:30:34.756 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:34 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:34.557+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:34.757 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:34 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:34.557+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:35.156 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:34 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:34.892+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:35.156 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:34 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:34.892+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:35.156 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:34 smithi181 podman[66887]: 2024-04-27 01:30:34.928615136 +0000 UTC m=+0.102170300 container create 6a30db3aa6a6378fb03f6fe708f6174bb81bbafc24979c84d0668f528c65ab3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=squid-b22e2eb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-27T01:30:35.156 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:34 smithi181 podman[66887]: 2024-04-27 01:30:34.844744539 +0000 UTC m=+0.018299704 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:30:35.156 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:35 smithi181 podman[66887]: 2024-04-27 01:30:35.105816683 +0000 UTC m=+0.279371847 container init 6a30db3aa6a6378fb03f6fe708f6174bb81bbafc24979c84d0668f528c65ab3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-b22e2eb) 2024-04-27T01:30:35.157 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:35 smithi181 podman[66887]: 2024-04-27 01:30:35.108950219 +0000 UTC m=+0.282505384 container start 6a30db3aa6a6378fb03f6fe708f6174bb81bbafc24979c84d0668f528c65ab3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:30:35.157 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:35 smithi181 podman[66887]: 2024-04-27 01:30:35.172427736 +0000 UTC m=+0.345982915 container attach 6a30db3aa6a6378fb03f6fe708f6174bb81bbafc24979c84d0668f528c65ab3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-27T01:30:35.506 INFO:journalctl@ceph.osd.4.smithi181.stdout:Apr 27 01:30:35 smithi181 podman[66887]: 2024-04-27 01:30:35.400827754 +0000 UTC m=+0.574382918 container died 6a30db3aa6a6378fb03f6fe708f6174bb81bbafc24979c84d0668f528c65ab3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, ceph=True, org.label-schema.build-date=20240424) 2024-04-27T01:30:35.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:35 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:35.591+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:35.902 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:35 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:35.591+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:35.902 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:35 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:35.597+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:35.902 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:35 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:35.597+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:36.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.4.service' 2024-04-27T01:30:36.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:30:36.066 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-27T01:30:36.066 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-27T01:30:36.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.5 2024-04-27T01:30:36.255 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:36 smithi181 systemd[1]: Stopping Ceph osd.5 for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:30:36.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:35 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:35.917+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:36.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:35 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:35.917+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:36.624 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:36.396+0000 7f05d476d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:30:36.624 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:36.396+0000 7f05d476d700 -1 osd.5 785 *** Got signal Terminated *** 2024-04-27T01:30:36.624 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:36.396+0000 7f05d476d700 -1 osd.5 785 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:30:36.624 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:36.550+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:36.624 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:36.550+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:36.917 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:36.639+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:36.918 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:36.639+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:37.219 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:36.933+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:37.220 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:36 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:36.933+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:37.961 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:37 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:37.675+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:37.961 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:37 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:37.675+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:37.962 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:37 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:37.598+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:37.962 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:37 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:37.598+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:38.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:37 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:37.977+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:38.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:37 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:37.977+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:38.963 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:38 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:38.627+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:38.963 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:38 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:38.627+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:38.963 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:38 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:38.616+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:38.963 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:38 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:38.616+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:39.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:38 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:38.978+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:39.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:38 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:38.978+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:39.940 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:39 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:39.674+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:39.941 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:39 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:39.674+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:39.941 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:39 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:39.599+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:39.941 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:39 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:39.599+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:40.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:39 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:39.956+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:40.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:39 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:39.956+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:40.928 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:40 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:40.716+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:57.772429+0000 front 2024-04-27T01:29:57.772645+0000 (oldest deadline 2024-04-27T01:30:25.972815+0000) 2024-04-27T01:30:40.928 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:40 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5[40788]: 2024-04-27T01:30:40.716+0000 7f05ccd59700 -1 osd.5 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:05.973437+0000 front 2024-04-27T01:30:05.973402+0000 (oldest deadline 2024-04-27T01:30:30.073231+0000) 2024-04-27T01:30:40.928 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:40 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:40.605+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:40.928 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:40 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:40.605+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:40.929 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:40 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:40.605+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:15.798177+0000 front 2024-04-27T01:30:15.798231+0000 (oldest deadline 2024-04-27T01:30:39.898221+0000) 2024-04-27T01:30:41.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:40 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:40.944+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:41.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:40 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:40.944+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:41.926 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:41 smithi181 podman[66993]: 2024-04-27 01:30:41.810859095 +0000 UTC m=+5.511747599 container died 90d6a2a517f65a8b7217b5733d55ceb786ae6781585df5b76be41db646426e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:30:41.926 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:41 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:41.593+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:41.926 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:41 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:41.593+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:41.927 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:41 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:41.593+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:15.798177+0000 front 2024-04-27T01:30:15.798231+0000 (oldest deadline 2024-04-27T01:30:39.898221+0000) 2024-04-27T01:30:42.188 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:42 smithi181 podman[66993]: 2024-04-27 01:30:42.187099003 +0000 UTC m=+5.887987520 container cleanup 90d6a2a517f65a8b7217b5733d55ceb786ae6781585df5b76be41db646426e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-27T01:30:42.188 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:42 smithi181 bash[66993]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5 2024-04-27T01:30:42.188 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:41 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:41.941+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:42.189 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:41 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:41.941+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:42.189 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:41 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:41.941+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:42.746 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:42 smithi181 podman[67021]: 2024-04-27 01:30:42.430183228 +0000 UTC m=+0.616077398 container remove 90d6a2a517f65a8b7217b5733d55ceb786ae6781585df5b76be41db646426e25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:30:42.746 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:42.568+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:42.747 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:42.568+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:42.747 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:42.568+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:15.798177+0000 front 2024-04-27T01:30:15.798231+0000 (oldest deadline 2024-04-27T01:30:39.898221+0000) 2024-04-27T01:30:43.005 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:42.916+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:43.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:42.916+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:43.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:42 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:42.916+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:43.006 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:42 smithi181 podman[67113]: 2024-04-27 01:30:42.665169888 +0000 UTC m=+0.018048638 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:30:43.006 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:42 smithi181 podman[67113]: 2024-04-27 01:30:42.764424829 +0000 UTC m=+0.117303565 container create d9dd8b120e7b4f0f8165a67cfc0fa8a4c34cf138f9fb6cd19e24c354413a5f93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-27T01:30:43.006 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:43 smithi181 podman[67113]: 2024-04-27 01:30:43.000077044 +0000 UTC m=+0.352955794 container init d9dd8b120e7b4f0f8165a67cfc0fa8a4c34cf138f9fb6cd19e24c354413a5f93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-27T01:30:43.006 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:43 smithi181 podman[67113]: 2024-04-27 01:30:43.003057772 +0000 UTC m=+0.355936505 container start d9dd8b120e7b4f0f8165a67cfc0fa8a4c34cf138f9fb6cd19e24c354413a5f93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-27T01:30:43.505 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:43 smithi181 podman[67113]: 2024-04-27 01:30:43.091887608 +0000 UTC m=+0.444766354 container attach d9dd8b120e7b4f0f8165a67cfc0fa8a4c34cf138f9fb6cd19e24c354413a5f93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-27T01:30:43.505 INFO:journalctl@ceph.osd.5.smithi181.stdout:Apr 27 01:30:43 smithi181 podman[67113]: 2024-04-27 01:30:43.291112787 +0000 UTC m=+0.643991515 container died d9dd8b120e7b4f0f8165a67cfc0fa8a4c34cf138f9fb6cd19e24c354413a5f93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-27T01:30:43.833 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:43 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:43.580Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-04-27T01:30:43.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:43 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:43.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:30:43.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:43 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:43.581Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-04-27T01:30:43.854 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:43 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:43.607+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:43.854 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:43 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:43.607+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:43.854 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:43 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:43.607+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:15.798177+0000 front 2024-04-27T01:30:15.798231+0000 (oldest deadline 2024-04-27T01:30:39.898221+0000) 2024-04-27T01:30:43.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.5.service' 2024-04-27T01:30:43.915 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:30:43.915 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-27T01:30:43.916 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-27T01:30:43.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.6 2024-04-27T01:30:44.224 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:44 smithi181 systemd[1]: Stopping Ceph osd.6 for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:30:44.224 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:43 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:43.895+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:44.224 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:43 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:43.895+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:44.224 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:43 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:43.895+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:44.505 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:44 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:44.239+0000 7f7f69bb8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:30:44.506 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:44 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:44.239+0000 7f7f69bb8700 -1 osd.6 785 *** Got signal Terminated *** 2024-04-27T01:30:44.506 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:44 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:44.239+0000 7f7f69bb8700 -1 osd.6 785 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:30:44.909 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:44 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:44.650+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:44.910 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:44 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:44.650+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:44.910 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:44 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:44.650+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:15.798177+0000 front 2024-04-27T01:30:15.798231+0000 (oldest deadline 2024-04-27T01:30:39.898221+0000) 2024-04-27T01:30:45.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:44 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:44.925+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:45.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:44 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:44.925+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:45.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:44 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:44.925+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:45.876 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:45 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:45.628+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:45.876 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:45 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:45.628+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:45.876 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:45 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:45.628+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:15.798177+0000 front 2024-04-27T01:30:15.798231+0000 (oldest deadline 2024-04-27T01:30:39.898221+0000) 2024-04-27T01:30:46.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:45 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:45.892+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:46.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:45 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:45.892+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:46.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:45 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:45.892+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:46.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:45 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:45.892+0000 7fc114869700 -1 osd.7 785 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14577.0:1208 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e785) 2024-04-27T01:30:47.005 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:46 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:46.617+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:47.006 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:46 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:46.617+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:47.006 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:46 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:46.617+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:15.798177+0000 front 2024-04-27T01:30:15.798231+0000 (oldest deadline 2024-04-27T01:30:39.898221+0000) 2024-04-27T01:30:47.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:46 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:46.857+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:47.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:46 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:46.857+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:47.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:46 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:46.857+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:47.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:46 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:46.858+0000 7fc114869700 -1 osd.7 785 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14577.0:1208 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e785) 2024-04-27T01:30:48.005 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:47 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:47.579+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:48.005 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:47 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:47.579+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:48.005 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:47 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:47.579+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:15.798177+0000 front 2024-04-27T01:30:15.798231+0000 (oldest deadline 2024-04-27T01:30:39.898221+0000) 2024-04-27T01:30:48.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:47 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:47.812+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:48.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:47 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:47.812+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:48.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:47 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:47.812+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:49.005 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:48 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:48.566+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:30:00.493951+0000 front 2024-04-27T01:30:00.494079+0000 (oldest deadline 2024-04-27T01:30:23.394101+0000) 2024-04-27T01:30:49.006 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:48 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:48.566+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:09.497043+0000 front 2024-04-27T01:30:09.497091+0000 (oldest deadline 2024-04-27T01:30:34.697245+0000) 2024-04-27T01:30:49.006 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:48 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6[46632]: 2024-04-27T01:30:48.566+0000 7f7f621a4700 -1 osd.6 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:15.798177+0000 front 2024-04-27T01:30:15.798231+0000 (oldest deadline 2024-04-27T01:30:39.898221+0000) 2024-04-27T01:30:49.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:48 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:48.784+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:49.006 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:48 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:48.784+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:49.007 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:48 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:48.784+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:49.991 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:49 smithi181 podman[67219]: 2024-04-27 01:30:49.680031405 +0000 UTC m=+5.533214448 container died 245cef1834d122c5c93748c912acffbead8f3bb237de48560089c3392aa3859a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:30:49.991 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:49 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:49.810+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:49.991 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:49 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:49.810+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:49.991 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:49 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:49.810+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:50.255 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:50 smithi181 podman[67219]: 2024-04-27 01:30:50.072731003 +0000 UTC m=+5.925914048 container cleanup 245cef1834d122c5c93748c912acffbead8f3bb237de48560089c3392aa3859a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:30:50.256 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:50 smithi181 bash[67219]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6 2024-04-27T01:30:50.675 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:50 smithi181 podman[67237]: 2024-04-27 01:30:50.340545756 +0000 UTC m=+0.654321260 container remove 245cef1834d122c5c93748c912acffbead8f3bb237de48560089c3392aa3859a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-27T01:30:50.936 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:50 smithi181 podman[67326]: 2024-04-27 01:30:50.691389365 +0000 UTC m=+0.112996145 container create 739038d328b9c34fd9f6b0d80e51b03d4bf1663abc652cc720937c238c463be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T01:30:50.936 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:50 smithi181 podman[67326]: 2024-04-27 01:30:50.596373835 +0000 UTC m=+0.017980570 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:30:50.936 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:50 smithi181 podman[67326]: 2024-04-27 01:30:50.868695673 +0000 UTC m=+0.290302412 container init 739038d328b9c34fd9f6b0d80e51b03d4bf1663abc652cc720937c238c463be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb) 2024-04-27T01:30:50.936 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:50 smithi181 podman[67326]: 2024-04-27 01:30:50.871727313 +0000 UTC m=+0.293334048 container start 739038d328b9c34fd9f6b0d80e51b03d4bf1663abc652cc720937c238c463be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0) 2024-04-27T01:30:50.936 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:50 smithi181 podman[67326]: 2024-04-27 01:30:50.951817389 +0000 UTC m=+0.373424135 container attach 739038d328b9c34fd9f6b0d80e51b03d4bf1663abc652cc720937c238c463be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-27T01:30:50.936 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:50.829+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:50.936 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:50.829+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:50.936 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:50 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:50.829+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:51.255 INFO:journalctl@ceph.osd.6.smithi181.stdout:Apr 27 01:30:51 smithi181 podman[67326]: 2024-04-27 01:30:51.162743803 +0000 UTC m=+0.584350545 container died 739038d328b9c34fd9f6b0d80e51b03d4bf1663abc652cc720937c238c463be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-04-27T01:30:51.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.6.service' 2024-04-27T01:30:51.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:30:51.820 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-27T01:30:51.820 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-27T01:30:51.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.7 2024-04-27T01:30:52.150 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:30:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:30:51.835Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.162:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.162:8765: connect: connection refused" 2024-04-27T01:30:52.151 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:30:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:30:51.836Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.162:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.162:8765: connect: connection refused" 2024-04-27T01:30:52.151 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:30:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:30:51.837Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.162:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.162:8765: connect: connection refused" 2024-04-27T01:30:52.151 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:30:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:30:51.837Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.162:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.162:8765: connect: connection refused" 2024-04-27T01:30:52.151 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:30:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:30:51.837Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.162:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.162:8765: connect: connection refused" 2024-04-27T01:30:52.151 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:51.860+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:52.151 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:51.860+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:52.151 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:51.860+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:52.151 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:51 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:51.860+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-04-27T01:30:21.922548+0000 front 2024-04-27T01:30:21.922519+0000 (oldest deadline 2024-04-27T01:30:51.322888+0000) 2024-04-27T01:30:52.151 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:51 smithi181 systemd[1]: Stopping Ceph osd.7 for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:30:52.151 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:52.166+0000 7fc11c27d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:30:52.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:52.166+0000 7fc11c27d700 -1 osd.7 785 *** Got signal Terminated *** 2024-04-27T01:30:52.152 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:52.166+0000 7fc11c27d700 -1 osd.7 785 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T01:30:53.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:52.885+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:53.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:52.885+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:53.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:52.885+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:53.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:52 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:52.885+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-04-27T01:30:21.922548+0000 front 2024-04-27T01:30:21.922519+0000 (oldest deadline 2024-04-27T01:30:51.322888+0000) 2024-04-27T01:30:53.833 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:53 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:53.581Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:30:53.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:53 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:53.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:30:53.834 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:30:53 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:30:53.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-04-27T01:30:54.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:53 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:53.926+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:54.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:53 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:53.926+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:54.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:53 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:53.926+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:54.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:53 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:53.926+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-04-27T01:30:21.922548+0000 front 2024-04-27T01:30:21.922519+0000 (oldest deadline 2024-04-27T01:30:51.322888+0000) 2024-04-27T01:30:55.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:54 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:54.888+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:55.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:54 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:54.888+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:55.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:54 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:54.888+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:55.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:54 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:54.888+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-04-27T01:30:21.922548+0000 front 2024-04-27T01:30:21.922519+0000 (oldest deadline 2024-04-27T01:30:51.322888+0000) 2024-04-27T01:30:56.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:55 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:55.916+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:56.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:55 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:55.916+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:56.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:55 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:55.916+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:56.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:55 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:55.916+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-04-27T01:30:21.922548+0000 front 2024-04-27T01:30:21.922519+0000 (oldest deadline 2024-04-27T01:30:51.322888+0000) 2024-04-27T01:30:57.255 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:56 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:56.940+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6803 osd.0 since back 2024-04-27T01:29:59.119625+0000 front 2024-04-27T01:29:59.119546+0000 (oldest deadline 2024-04-27T01:30:22.619645+0000) 2024-04-27T01:30:57.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:56 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:56.940+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6807 osd.1 since back 2024-04-27T01:30:08.421014+0000 front 2024-04-27T01:30:08.421073+0000 (oldest deadline 2024-04-27T01:30:34.321144+0000) 2024-04-27T01:30:57.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:56 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:56.940+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6811 osd.2 since back 2024-04-27T01:30:16.021918+0000 front 2024-04-27T01:30:16.021873+0000 (oldest deadline 2024-04-27T01:30:41.921913+0000) 2024-04-27T01:30:57.256 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:56 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7[52694]: 2024-04-27T01:30:56.940+0000 7fc114869700 -1 osd.7 785 heartbeat_check: no reply from 172.21.15.162:6815 osd.3 since back 2024-04-27T01:30:21.922548+0000 front 2024-04-27T01:30:21.922519+0000 (oldest deadline 2024-04-27T01:30:51.322888+0000) 2024-04-27T01:30:57.755 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:57 smithi181 podman[67430]: 2024-04-27 01:30:57.511723847 +0000 UTC m=+5.449783523 container died 0fb1603cd4a197e9cf474040e51076fe90ace041ffe9139d1482c9bd77fb6f5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T01:30:58.109 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:57 smithi181 podman[67430]: 2024-04-27 01:30:57.8740039 +0000 UTC m=+5.812063585 container cleanup 0fb1603cd4a197e9cf474040e51076fe90ace041ffe9139d1482c9bd77fb6f5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, org.label-schema.schema-version=1.0) 2024-04-27T01:30:58.109 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:57 smithi181 bash[67430]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7 2024-04-27T01:30:58.428 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:58 smithi181 podman[67449]: 2024-04-27 01:30:58.125006693 +0000 UTC m=+0.616711273 container remove 0fb1603cd4a197e9cf474040e51076fe90ace041ffe9139d1482c9bd77fb6f5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-b22e2eb, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True) 2024-04-27T01:30:58.737 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:58 smithi181 podman[67545]: 2024-04-27 01:30:58.347799813 +0000 UTC m=+0.018301417 image pull 87e5280d5ae87e34552d9b448225db981ca5313b28020853b569ec2cb8ee53ae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6 2024-04-27T01:30:58.737 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:58 smithi181 podman[67545]: 2024-04-27 01:30:58.450693768 +0000 UTC m=+0.121195409 container create b35cb620e52ad26a49d47218f3c432450b65414d4f77adad97384ce3547c6daa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-b22e2eb, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-27T01:30:58.737 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:58 smithi181 podman[67545]: 2024-04-27 01:30:58.669762115 +0000 UTC m=+0.340263710 container init b35cb620e52ad26a49d47218f3c432450b65414d4f77adad97384ce3547c6daa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=squid-b22e2eb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-04-27T01:30:58.737 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:58 smithi181 podman[67545]: 2024-04-27 01:30:58.672753477 +0000 UTC m=+0.343255080 container start b35cb620e52ad26a49d47218f3c432450b65414d4f77adad97384ce3547c6daa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb) 2024-04-27T01:30:58.738 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:58 smithi181 podman[67545]: 2024-04-27 01:30:58.752739756 +0000 UTC m=+0.423241371 container attach b35cb620e52ad26a49d47218f3c432450b65414d4f77adad97384ce3547c6daa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=squid-b22e2eb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True) 2024-04-27T01:30:59.005 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:58 smithi181 podman[67545]: 2024-04-27 01:30:58.964676913 +0000 UTC m=+0.635178511 container died b35cb620e52ad26a49d47218f3c432450b65414d4f77adad97384ce3547c6daa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-b22e2eb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-27T01:30:59.486 INFO:journalctl@ceph.osd.7.smithi181.stdout:Apr 27 01:30:59 smithi181 podman[67571]: 2024-04-27 01:30:59.480894225 +0000 UTC m=+0.510325400 container remove b35cb620e52ad26a49d47218f3c432450b65414d4f77adad97384ce3547c6daa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-b22e2eb, maintainer=Guillaume Abrioux ) 2024-04-27T01:30:59.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@osd.7.service' 2024-04-27T01:30:59.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:30:59.527 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-27T01:30:59.527 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-04-27T01:30:59.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@rgw.foo.a 2024-04-27T01:30:59.824 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:30:59 smithi162 systemd[1]: Stopping Ceph rgw.foo.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:31:00.083 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:30:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a[63945]: 2024-04-27T01:30:59.840+0000 7f0332313700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T01:31:00.083 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:30:59 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a[63945]: 2024-04-27T01:30:59.840+0000 7f0340b91ac0 -1 shutting down 2024-04-27T01:31:03.833 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:03 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:31:03.582Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:31:03.833 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:03 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:31:03.582Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:31:03.833 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:03 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:31:03.583Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-04-27T01:31:10.083 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:31:09 smithi162 bash[147548]: time="2024-04-27T01:31:09Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-04-27T01:31:10.083 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:31:09 smithi162 podman[147548]: 2024-04-27 01:31:09.860440728 +0000 UTC m=+10.109563481 container died b6dba62f254e72e3e9b351c6e9960e83597ca043608eb07b1ba02c1a99464456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-27T01:31:10.348 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:31:10 smithi162 podman[147548]: 2024-04-27 01:31:10.196805718 +0000 UTC m=+10.445928472 container cleanup b6dba62f254e72e3e9b351c6e9960e83597ca043608eb07b1ba02c1a99464456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-b22e2eb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T01:31:10.348 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:31:10 smithi162 bash[147548]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a 2024-04-27T01:31:10.604 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:31:10 smithi162 podman[147564]: 2024-04-27 01:31:10.364526901 +0000 UTC m=+0.499297928 container remove b6dba62f254e72e3e9b351c6e9960e83597ca043608eb07b1ba02c1a99464456 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-rgw-foo-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=squid-b22e2eb, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-27T01:31:10.604 INFO:journalctl@ceph.rgw.foo.a.smithi162.stdout:Apr 27 01:31:10 smithi162 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-04-27T01:31:10.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@rgw.foo.a.service' 2024-04-27T01:31:10.643 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:31:10.643 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-04-27T01:31:10.643 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-27T01:31:10.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ceph-ea378f3c-0431-11ef-bc93-c7b262605968@prometheus.a 2024-04-27T01:31:10.941 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 systemd[1]: Stopping Ceph prometheus.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.957Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.957Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.957Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.957Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.957Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.957Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.957Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.957Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.957Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.958Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.958Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-27T01:31:11.201 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a[62310]: ts=2024-04-27T01:31:10.958Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-27T01:31:11.202 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:10 smithi181 podman[67652]: 2024-04-27 01:31:10.967929182 +0000 UTC m=+0.103025440 container died 8b221f51564304e296a84629833ab84319976c265bc591502cef6741287c06eb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-27T01:31:11.505 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:11 smithi181 podman[67652]: 2024-04-27 01:31:11.33222594 +0000 UTC m=+0.467322198 container cleanup 8b221f51564304e296a84629833ab84319976c265bc591502cef6741287c06eb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-27T01:31:11.505 INFO:journalctl@ceph.prometheus.a.smithi181.stdout:Apr 27 01:31:11 smithi181 bash[67652]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-prometheus-a 2024-04-27T01:31:11.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ea378f3c-0431-11ef-bc93-c7b262605968@prometheus.a.service' 2024-04-27T01:31:11.734 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T01:31:11.734 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-27T01:31:11.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm rm-cluster --fsid ea378f3c-0431-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T01:31:11.923 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:31:13.583 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:13 smithi162 systemd[1]: Stopping Ceph alertmanager.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:31:13.583 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:13 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:31:13.583Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:31:13.584 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:13 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:31:13.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi162:8443/api/prometheus_receiver\": dial tcp 172.21.15.162:8443: connect: connection refused" 2024-04-27T01:31:13.584 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:13 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:31:13.584Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi181.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.181:8443: connect: connection refused" 2024-04-27T01:31:14.083 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:13 smithi162 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a[73581]: ts=2024-04-27T01:31:13.767Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-27T01:31:14.083 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:13 smithi162 podman[148045]: 2024-04-27 01:31:13.818716597 +0000 UTC m=+0.166320291 container died a798feaa6f73ad454dd68307082716b40072e035e27e34c22e66e1ec9089fff7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:31:14.427 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:14 smithi162 podman[148045]: 2024-04-27 01:31:14.184252013 +0000 UTC m=+0.531855710 container cleanup a798feaa6f73ad454dd68307082716b40072e035e27e34c22e66e1ec9089fff7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:31:14.427 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:14 smithi162 bash[148045]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a 2024-04-27T01:31:14.829 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:14 smithi162 podman[148059]: 2024-04-27 01:31:14.443636897 +0000 UTC m=+0.629624625 container remove a798feaa6f73ad454dd68307082716b40072e035e27e34c22e66e1ec9089fff7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-27T01:31:14.829 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:14 smithi162 podman[148059]: 2024-04-27 01:31:14.5110294 +0000 UTC m=+0.697017085 volume remove d32d3a76b94a4ea7dc935ba9bb1bc2c775e14569cd4bb47ac0649b4c8b1b713b 2024-04-27T01:31:14.829 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:14 smithi162 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-27T01:31:14.829 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:14 smithi162 systemd[1]: Stopped Ceph alertmanager.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:31:14.830 INFO:journalctl@ceph.alertmanager.a.smithi162.stdout:Apr 27 01:31:14 smithi162 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@alertmanager.a.service: Consumed 1.725s CPU time. 2024-04-27T01:31:15.083 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:31:14 smithi162 systemd[1]: Stopping Ceph node-exporter.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:31:15.083 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:31:15 smithi162 podman[148172]: 2024-04-27 01:31:15.090711012 +0000 UTC m=+0.107143693 container died f34e1c8822b17aa863c70511f51f8c4a77a04c23a8a8374574fdc8558d57131c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-27T01:31:15.583 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:31:15 smithi162 podman[148172]: 2024-04-27 01:31:15.396544078 +0000 UTC m=+0.412976767 container cleanup f34e1c8822b17aa863c70511f51f8c4a77a04c23a8a8374574fdc8558d57131c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-27T01:31:15.583 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:31:15 smithi162 bash[148172]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a 2024-04-27T01:31:15.895 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:31:15 smithi162 podman[148186]: 2024-04-27 01:31:15.622294617 +0000 UTC m=+0.532377472 container remove f34e1c8822b17aa863c70511f51f8c4a77a04c23a8a8374574fdc8558d57131c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-27T01:31:15.895 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:31:15 smithi162 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-27T01:31:15.895 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:31:15 smithi162 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-27T01:31:15.895 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:31:15 smithi162 systemd[1]: Stopped Ceph node-exporter.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:31:15.895 INFO:journalctl@ceph.node-exporter.a.smithi162.stdout:Apr 27 01:31:15 smithi162 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@node-exporter.a.service: Consumed 3.494s CPU time. 2024-04-27T01:31:16.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm rm-cluster --fsid ea378f3c-0431-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T01:31:16.567 INFO:teuthology.orchestra.run.smithi181.stdout:Deleting cluster with fsid: ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:31:18.005 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:17 smithi181 systemd[1]: Stopping Ceph iscsi.iscsi.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:31:18.505 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:18 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a[57504]: debug Shutdown received 2024-04-27T01:31:28.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:28 smithi181 bash[68077]: time="2024-04-27T01:31:28Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-04-27T01:31:28.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:28 smithi181 podman[68077]: 2024-04-27 01:31:28.093359374 +0000 UTC m=+10.115595295 container died bafd375641d6267be09baa78894f863e69acbdef858f24f2040c62f56d106c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-b22e2eb) 2024-04-27T01:31:28.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:28 smithi181 podman[68077]: 2024-04-27 01:31:28.443847332 +0000 UTC m=+10.466083253 container cleanup bafd375641d6267be09baa78894f863e69acbdef858f24f2040c62f56d106c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-27T01:31:28.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:28 smithi181 bash[68077]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a 2024-04-27T01:31:29.005 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:28 smithi181 podman[68094]: 2024-04-27 01:31:28.694790375 +0000 UTC m=+0.601860152 container remove bafd375641d6267be09baa78894f863e69acbdef858f24f2040c62f56d106c94 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:265587011b499c966b9dbaf00ce5756e85851e17df544df961d9e3203c8e39d6, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-b22e2eb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T01:31:29.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:28 smithi181 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-04-27T01:31:29.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:28 smithi181 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-04-27T01:31:29.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:28 smithi181 systemd[1]: Stopped Ceph iscsi.iscsi.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:31:29.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi181.stdout:Apr 27 01:31:28 smithi181 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@iscsi.iscsi.a.service: Consumed 3.436s CPU time. 2024-04-27T01:31:29.564 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:29 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=cleanup t=2024-04-27T01:31:29.291203342Z level=info msg="Completed cleanup jobs" duration=6.438982ms 2024-04-27T01:31:30.505 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:30 smithi181 systemd[1]: Stopping Ceph grafana.a for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:31:30.930 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=server t=2024-04-27T01:31:30.671573909Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-04-27T01:31:30.930 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=tracing t=2024-04-27T01:31:30.671772209Z level=info msg="Closing tracing" 2024-04-27T01:31:30.930 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:30 smithi181 ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a[60968]: logger=ticker t=2024-04-27T01:31:30.671791604Z level=info msg=stopped last_tick=2024-04-27T01:31:30Z 2024-04-27T01:31:30.930 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:30 smithi181 podman[68356]: 2024-04-27 01:31:30.681714549 +0000 UTC m=+0.102436164 container died 8811dd0e2dd43ec89a3670da8d322d304ec7ba48b980259409f8d80d8b32ba25 (image=quay.io/ceph/grafana:9.4.12, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-27T01:31:31.247 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:31 smithi181 podman[68356]: 2024-04-27 01:31:31.029073961 +0000 UTC m=+0.449795563 container cleanup 8811dd0e2dd43ec89a3670da8d322d304ec7ba48b980259409f8d80d8b32ba25 (image=quay.io/ceph/grafana:9.4.12, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-27T01:31:31.247 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:31 smithi181 bash[68356]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a 2024-04-27T01:31:31.505 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:31 smithi181 podman[68370]: 2024-04-27 01:31:31.263487218 +0000 UTC m=+0.583904192 container remove 8811dd0e2dd43ec89a3670da8d322d304ec7ba48b980259409f8d80d8b32ba25 (image=quay.io/ceph/grafana:9.4.12, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-27T01:31:31.505 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:31 smithi181 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-27T01:31:31.505 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:31 smithi181 systemd[1]: Stopped Ceph grafana.a for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:31:31.505 INFO:journalctl@ceph.grafana.a.smithi181.stdout:Apr 27 01:31:31 smithi181 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@grafana.a.service: Consumed 13.652s CPU time. 2024-04-27T01:31:32.005 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:31:31 smithi181 systemd[1]: Stopping Ceph node-exporter.b for ea378f3c-0431-11ef-bc93-c7b262605968... 2024-04-27T01:31:32.005 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:31:31 smithi181 podman[68483]: 2024-04-27 01:31:31.826843115 +0000 UTC m=+0.099475065 container died 435a34b6823d6e76b0dedafcd34657b2a6ebe7406265a385463c702b3bd82d85 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-27T01:31:32.267 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:31:32 smithi181 podman[68483]: 2024-04-27 01:31:32.099345246 +0000 UTC m=+0.371977194 container cleanup 435a34b6823d6e76b0dedafcd34657b2a6ebe7406265a385463c702b3bd82d85 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-27T01:31:32.268 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:31:32 smithi181 bash[68483]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b 2024-04-27T01:31:32.651 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:31:32 smithi181 podman[68497]: 2024-04-27 01:31:32.28406159 +0000 UTC m=+0.461686744 container remove 435a34b6823d6e76b0dedafcd34657b2a6ebe7406265a385463c702b3bd82d85 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ea378f3c-0431-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-27T01:31:32.651 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:31:32 smithi181 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-27T01:31:32.651 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:31:32 smithi181 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-27T01:31:32.651 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:31:32 smithi181 systemd[1]: Stopped Ceph node-exporter.b for ea378f3c-0431-11ef-bc93-c7b262605968. 2024-04-27T01:31:32.651 INFO:journalctl@ceph.node-exporter.b.smithi181.stdout:Apr 27 01:31:32 smithi181 systemd[1]: ceph-ea378f3c-0431-11ef-bc93-c7b262605968@node-exporter.b.service: Consumed 3.493s CPU time. 2024-04-27T01:31:33.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:31:33.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T01:31:33.187 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-27T01:31:33.189 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458/remote/smithi162/crash 2024-04-27T01:31:33.190 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/crash -- . 2024-04-27T01:31:33.277 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T01:31:33.277 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-04-27T01:31:33.281 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458/remote/smithi181/crash 2024-04-27T01:31:33.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/crash -- . 2024-04-27T01:31:33.316 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T01:31:33.316 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now 2024-04-27T01:31:33.317 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-27T01:31:33.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-04-27T01:31:33.349 INFO:tasks.cephadm:Compressing logs... 2024-04-27T01:31:33.350 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T01:31:33.393 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T01:31:33.416 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.log 2024-04-27T01:31:33.417 INFO:teuthology.orchestra.run.smithi181.stderr:: No such file or directory 2024-04-27T01:31:33.417 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T01:31:33.417 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-27T01:31:33.417 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-27T01:31:33.417 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-27T01:31:33.417 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-27T01:31:33.417 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.5.log: /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-27T01:31:33.420 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-27T01:31:33.420 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.log 2024-04-27T01:31:33.420 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-27T01:31:33.420 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T01:31:33.420 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-27T01:31:33.420 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-27T01:31:33.420 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-27T01:31:33.420 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-27T01:31:33.420 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-27T01:31:33.421 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T01:31:33.421 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/tcmu-runner.log 2024-04-27T01:31:33.421 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.cephadm.log: 81.7% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T01:31:33.421 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T01:31:33.421 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/tcmu-runner.log: 66.9% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/tcmu-runner.log.gz 2024-04-27T01:31:33.421 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-27T01:31:33.421 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-volume.log: 87.0% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T01:31:33.425 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-27T01:31:33.425 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mon.c.log: 92.7% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T01:31:33.425 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T01:31:33.427 INFO:teuthology.orchestra.run.smithi181.stderr: 92.2% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T01:31:33.427 INFO:teuthology.orchestra.run.smithi181.stderr: 91.8% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-27T01:31:33.427 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T01:31:33.432 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 90.9% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T01:31:33.437 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mon.a.log: 95.2% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T01:31:33.441 INFO:teuthology.orchestra.run.smithi181.stderr: 91.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T01:31:33.444 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T01:31:33.445 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.cephadm.log: 89.1% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T01:31:33.460 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-client.rgw.foo.a.log 2024-04-27T01:31:33.470 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T01:31:33.488 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-client.rgw.foo.a.log: /var/log/ceph/cephadm.log: 90.9% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T01:31:33.499 INFO:teuthology.orchestra.run.smithi162.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T01:31:33.574 INFO:teuthology.orchestra.run.smithi162.stderr: 92.9% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-04-27T01:31:33.833 INFO:teuthology.orchestra.run.smithi162.stderr: 90.1% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-27T01:31:34.677 INFO:teuthology.orchestra.run.smithi162.stderr: 92.1% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-27T01:31:35.262 INFO:teuthology.orchestra.run.smithi181.stderr: 92.6% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-27T01:31:36.089 INFO:teuthology.orchestra.run.smithi162.stderr: 91.8% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-27T01:31:39.470 INFO:teuthology.orchestra.run.smithi181.stderr: 93.7% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-27T01:31:40.339 INFO:teuthology.orchestra.run.smithi181.stderr: 93.9% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-27T01:31:40.386 INFO:teuthology.orchestra.run.smithi181.stderr: 93.8% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-27T01:31:40.436 INFO:teuthology.orchestra.run.smithi181.stderr: 93.8% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-27T01:31:40.438 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-04-27T01:31:40.438 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m7.032s 2024-04-27T01:31:40.438 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m27.838s 2024-04-27T01:31:40.438 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.984s 2024-04-27T01:31:40.459 INFO:teuthology.orchestra.run.smithi162.stderr: 93.9% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-27T01:31:40.696 INFO:teuthology.orchestra.run.smithi162.stderr: 93.8% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-27T01:31:40.814 INFO:teuthology.orchestra.run.smithi162.stderr: 93.9% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-27T01:31:41.339 INFO:teuthology.orchestra.run.smithi162.stderr: 93.9% -- replaced with /var/log/ceph/ea378f3c-0431-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-27T01:31:41.342 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-04-27T01:31:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m7.936s 2024-04-27T01:31:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:user 0m32.767s 2024-04-27T01:31:41.342 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m1.178s 2024-04-27T01:31:41.342 INFO:tasks.cephadm:Archiving logs... 2024-04-27T01:31:41.342 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458/remote/smithi162/log 2024-04-27T01:31:41.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T01:31:45.616 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458/remote/smithi181/log 2024-04-27T01:31:45.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T01:31:49.643 INFO:tasks.cephadm:Removing cluster... 2024-04-27T01:31:49.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo cephadm rm-cluster --fsid ea378f3c-0431-11ef-bc93-c7b262605968 --force 2024-04-27T01:31:49.833 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:31:50.118 DEBUG:teuthology.orchestra.run.smithi181:> sudo cephadm rm-cluster --fsid ea378f3c-0431-11ef-bc93-c7b262605968 --force 2024-04-27T01:31:50.309 INFO:teuthology.orchestra.run.smithi181.stdout:Deleting cluster with fsid: ea378f3c-0431-11ef-bc93-c7b262605968 2024-04-27T01:31:50.596 INFO:tasks.cephadm:Teardown complete 2024-04-27T01:31:50.596 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-27T01:31:50.605 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-27T01:31:50.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T01:31:50.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T01:31:50.677 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-27T01:31:50.686 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-27T01:31:50.686 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T01:31:50.688 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T01:31:50.700 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-04-27T01:31:50.702 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-04-27T01:31:50.704 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T01:31:50.704 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-04-27T01:31:50.704 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv01.front.sepia.ceph.com 3 6 377 48 +1331us[+1396us] +/- 67ms 2024-04-27T01:31:50.704 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 47 -2008us[-2008us] +/- 62ms 2024-04-27T01:31:50.705 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 +939us[ +939us] +/- 72ms 2024-04-27T01:31:50.705 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:31:50.706 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T01:31:50.706 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-04-27T01:31:50.706 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 47 +1353us[+1442us] +/- 67ms 2024-04-27T01:31:50.706 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -1991us[-1901us] +/- 62ms 2024-04-27T01:31:50.706 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +964us[+1053us] +/- 72ms 2024-04-27T01:31:50.707 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T01:31:50.707 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-27T01:31:50.717 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-27T01:31:50.719 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-27T01:31:50.748 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T01:31:50.816 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T01:31:50.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-27T01:31:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:31:50.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-27T01:31:50.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:31:50.935 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-27T01:31:50.944 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-27T01:31:51.017 INFO:teuthology.task.internal:Duration was 2147.947730 seconds 2024-04-27T01:31:51.018 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-27T01:31:51.048 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-27T01:31:51.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T01:31:51.051 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T01:31:51.085 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T01:31:51.087 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T01:31:51.373 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-27T01:31:51.373 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-04-27T01:31:51.374 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T01:31:51.435 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-04-27T01:31:51.436 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T01:31:51.468 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-27T01:31:51.468 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T01:31:51.478 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T01:31:51.533 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-27T01:31:51.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T01:31:51.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T01:31:51.942 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-27T01:31:51.952 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-27T01:31:51.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T01:31:51.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T01:31:52.000 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-27T01:31:52.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T01:31:52.019 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T01:31:52.038 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-04-27T01:31:52.062 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-04-27T01:31:52.098 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T01:31:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:31:52.113 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T01:31:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T01:31:52.155 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-27T01:31:52.165 INFO:teuthology.task.internal:Transferring archived files... 2024-04-27T01:31:52.165 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458/remote/smithi162 2024-04-27T01:31:52.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T01:31:52.226 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_21:08:03-orch-squid-distro-default-smithi/7675458/remote/smithi181 2024-04-27T01:31:52.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T01:31:52.263 INFO:teuthology.task.internal:Removing archive directory... 2024-04-27T01:31:52.263 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T01:31:52.265 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T01:31:52.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-27T01:31:52.326 INFO:teuthology.task.internal:Not uploading archives. 2024-04-27T01:31:52.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-27T01:31:52.359 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-27T01:31:52.359 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T01:31:52.361 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T01:31:52.373 INFO:teuthology.orchestra.run.smithi162.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 01:31 /home/ubuntu/cephtest 2024-04-27T01:31:52.375 INFO:teuthology.orchestra.run.smithi181.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 01:31 /home/ubuntu/cephtest 2024-04-27T01:31:52.376 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-27T01:31:52.392 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-27T01:31:52.462 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} duration: 2147.9477303028107 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-27T01:31:52.462 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T01:31:52.545 INFO:teuthology.run:pass